🚀 Check out this trending post from Hacker News 📖
📂 **Category**:
✅ **What You’ll Learn**:
Interest in supporting the
MIPS
architecture in BSD is about as old as the architecture itself, and
Risc/OS
(the Unix variant running on MIPS’ own workstations, before MIPS got bought by
SGI and stopped manufacturing anything but processors)
was based upon BSD code; so was
Ultrix, Digital’s first
flavour of Unix, which ran on VAX but also on its MIPS-based
DECstations.
While there is a lot to tell on OpenBSD on SGI hardware alone, I think it is
better to see a larger picture. Be warned that this story is quite long!
[skip to the next part]
Introduction
Before I start narrating the tale of BSD on MIPS in general and of OpenBSD
on SGI in particular, you might want to familiarize yourself with SGI hardware
over the years.
An excellent resource about SGI systems, from the early Motorola 68000-based
IRISes to the latest Itanium-based Altix, can be found on Gerhard Lenerz’
sgistuff site.
In this narration, I will focus on only a subset of the MIPS-based systems.
You might want to keep an eye on the
timeline
page as well.
In the beginning of this story, the most iconic SGI systems are the
Indigo (the Song and Dance machine, as was silkscreened on some
of its boards), first released in 1991, which is getting replaced by the
Indy, SGI’s first workstation in pizzabox form factor, and the
Indigo2, a larger desktop case which can also be put deskside
with special feet, and has more expansion capabilities than the Indy.
All these three systems have nice coloured cases, with a dark blue (well…
indigo) for the Indigo, a clear blue for the Indy, and a clear teal for the
Indigo2 (some later Indigo2 models will sport purple
cases). This is quite the eye catcher, in a world where workstations tended to
be greyish (with some subtle lavender touches on Sun Aurora case used by the
SPARCstation 20, 5 and 4) or beige, just like the average PC clone case.
Another distincting feature of the SGI systems is that, when powered on, they
would play a short tune, different on every model.
(You can find
decent recordings
of some of these sounds on YouTube.
Interestingly, despite not having an internal speaker, the Octane nevertheless
has its own boot tune, but to hear it, you need to plug external speakers or
headphones into the rear jack.)
This made these workstations really appealing. Which is the #1 reason why
people have wanted to run BSD or Linux on them. Not only because it could be
done, but because these machines were the raddest.
In late 1996, the O2 was introduced to replace the Indy as the
entry-level workstation, and shortly after, in early 1997, the Octane was
introduced to replace the Indigo2. Both systems would now use PCI
cards for expansion, with the O2 having one PCI slot for a short card, and the
Octane having an optional PCI cardcage allowing for up to 3 cards of any length.
Both these systems carried on with the coloured case tradition, the
O2 being
dark blue, and the Octane green, but one surprising feature was that the top of
their cases were rounded, and it was no longer possible to put anything on them
(and definitely no 21″ CRT monitor, despite the Octane chassis being strong
enough to support its weight.) The Octane, despite being larger and wider than
the O2, had no internal 5″1/4 storage device bay, requiring external SCSI CD-ROM
drives, while the O2 came with a CD-ROM drive at the top of the case.
The Octane was also a multiprocessor capable machine, allowing for up to two
processors; customers needing more could buy the Origin 200, introduced
at about the same time; each Origin 200 system would have two processors, but
two Origin 200 systems could be connected with a so-called
NUMAlink cable,
to create a four processor machine. And if you had needs for even more
horsepower, the Origin 2000 family could easily allow for up to 128
processors, as long as you could afford them (and the electricity bill.)
(SGI marketing department even had five Octane-related songs recorded and
burned onto an audio CD given away to prospective customers; these songs are not
that bad and will probably make you smile. You can find decent quality mp3 files
for them
here.)
The Origin 2000 and 200 families were improved with the Origin 3000 and
Origin 300 lines in 2000 and 2001, with twice as many processors per node
(a single Origin 300 node would now have four processors). Then in 2002, these
were improved again with the Origin 350 and Origin 3500 families,
and the single-processor Fuel system intended to replace the Octane
(although, with one processor and only up to 4GB of memory, it was behind the
best Octane configurations, which could use two processors and 8GB of memory),
and some reliability problems in the early models quickly earned it the
Fool nickname.
The Fuel systems also stood apart from the rest of the SGI systems because of
their case colour; they were initially supposed to sport a nice blue colour, and
the Fuel prototypes did;
but once marketing settled on the name Fuel
(the codename for that machine had been Asterix until then), it
was obvious the case colour had to match with the implied fire capability of
the name, and it was changed to a bright red, which would remind old-timers of
the SGI
Crimson.
This is a very quick description of the SGI workstations, and to sum things up:
- in 1995, we wanted to run BSD on Indy, and wouldn’t it be nice if it also
ran on Indigo2…
…and we expected this to become a reality a few years later. - from 1997 onwards, we wanted to run BSD on O2, and wouldn’t it
be nice if it also ran on Octane…
…and we hoped this would become a reality a few years later. - from 2002 onwards, we wanted to run BSD on Fuel, and wouldn’t it be nice if
it also ran on Tezro and the rest of the Origin 350 family…
…and everyone agreed this was too much of a dream and would never happen.
But eventually, OpenBSD ran on
(almost) all these machines.
1988-1993, BSD
The first bits of MIPS code were written by Ralph Campbell of
the University of Utah, starting in 1988, but weren’t contributed to the
CSRG
at the University of Berkeley until early january 1992, where Kirk
McKusick
commited
the code with a simple “contributed by Ralph Campbell” log message.
This code was attempting to get BSD to run on the original Digital DECstations,
models 2100 and 3100, also known as pmax. In the beginning, this was
just the kernel, with the userland being the Ultrix binaries. But work on native
userland binaries followed, with the first commits occurring that same year
on
february 29th, by Ralph Campbell himself, who had received access to the
CSRG source code repository.
The code slowly matured, with more models of the DECstation 5000 series becoming
supported, and Rick Macklem contributed support for the
Personal DECstation.
However, there was no native toolchain for BSD on MIPS; it relied upon the
Ultrix cc, as and ld binaries running in emulation.
In parallel, building on that MIPS support, Japanese hackers, led by Kazumasa
Utashiro, were working on BSD support for the MIPS-based
Sony NEWS workstations;
this work got eventually commited to the CSRG source tree on
march
1993 as a news3400 port.
Here is what sys/news3400/README had to say:
# @(#)README 8.1 (Berkeley) 06/10/93
------------------------------------------------------------------------------
This Sony MIPS R3000 based workstation support is done as an activity of
WIDE research project. Sony Corporation contributed most device drivers
and gave us great technical supports. Kazumasa Utashiro worked mainly
4.4BSD implementation but that work was based on 4.3BSD-Reno port helped by
other project members, especially Tadamichi Matsuyama and Hidetoshi Unno.
This version is just a snapshot of developing system and has many
unimplemented features and bugs. Please contact utashiro@sra.co.jp if you
have any comments about this code. Bugfix will be greatly appreciated.
------------------------------------------------------------------------------
I've been using NWS-3200 laptop machine and NWS-3400 for development.
3200's LCD display and NWB-253 display board are supported now.
------------------------------------------------------------------------------
Currently GCC is used as a compiler which uses as and ld come with NEWS-OS.
They should soon be replaced by the latest GAS because it supports BSD
a.out format for MIPS.
------------------------------------------------------------------------------
Known bugs:
- Display driver is slow.
- Serial line driver is slow.
- Console doesn't accept ^S.
- Some problem probably in VM.
------------------------------------------------------------------------------
Kazumasa Utashiro
Software Research Associates, Inc.
1-1-1 Hirakawa-cho, Chiyoda-ku, Tokyo 102, Japan
and
WIDE Project
(that work, much later, would be reused and improved and become the
NetBSD/newsmips port.)
On the
16th june
of 1993, Kirk McKusick commited a “final update from Ralph”,
and not much happened on the MIPS front until Ralph Campbell put
“final changes for pmax 4.4-Lite II release” in
early
june 1995.
In the meantime, the NetBSD project had been created in 1993, with the goal of
running on as many hardware platforms as possible. A port-pmax
dedicated mailinglist was created in november 1993, and among the first messages
on this list, Theo de Raadt, one of the founders of the NetBSD project
and still involved in NetBSD at that time, wrote that he would
“love a
mips port”.
From: Theo de Raadt To: port-pmax Subject: Re: General status information Date: 11/22/1993 19:15:44 > What's the status of the port? I've got a few spare DS2100 or better > class machines sitting around... The code is in the tree almost unchanged from how it was donated. It needs to be fixed to do things "the NetBSD way". A few people volunteered to help get this code working, but we've not heard much from them. The code is from 4.4 -- hence it is much easier/better to merge it into the NetBSD magnum branch. This branch is (as yet) not available to the net at large... only people who have sun-lamp accounts can work on it easily. The magnum branch does various things as they are done in 4.4. These things ARE important -- it would probably take 4-6 times longer to make the pmax code work in a NetBSD-current environment. A number of ports already use the magnum tree (sparc, sun3) and others are being changed to use it (i386, pc532). Yes.. that means there are about 4 people familiar with what changes need to be made. So, if you're a wiley hacker who wants to make this happen, get in touch with me. (It should not take too long if you pay close attention to what had to be changed to make the sparc port fit in, perhaps 2 weeks till you are able to build a crash-and-burn kernel.) I would *love* to see this port working.
1994, NetBSD
However, a lot of work was required to make the port self-hosting, which was
a strong requirement for NetBSD. Not much development happened behind the
scenes, and the mailinglist was quite quiet, leading to Ralph Campbell himself
(no longer at the University of Utah) asking for a status update early in
may
1994.
This led to an answer from Adam Glass (another NetBSD co-founder)
giving an
optimistic
timeline.
By the middle of june, the port had made
significant
progress.
Work progressed in order to get a native toolchain (based on gcc 2.5.8 and
heavily modified binutils 2.4 at that time), but by the time NetBSD 1.0
was released, on november 8th, pmax was not yet an official part of the release.
Then Per Fogelström (not the swedish author!) appeared out of nowhere,
with an
EPIC
message, revealing that he was in fact no less than the
Chuck Norris
of the MIPS world.
From: Per Fogelström
To: port-pmax
Subject: NetBSD Mips port, Success story.
Date: 12/10/1994 18:22:05
Hi everybody!
There have been much talk about the pmax port during the last days.
I'm not a pmax owner, have no funds, so i built my own system.
Porting NetBSD to it was quite stright forward so i don't recognize
all the problems you describe. Perhaps many of them arise from using
Ultrix as a porting plattform. :-> Or at least, using Ultrix C compiler
and tools is giving you problems.
Anyway i have a R3000 system running native without major problems.
This is the story how it was done. Maybe it can give you some ideas
or at least entertain you for a couple of minutes.
HOW NetBSD WAS PORTED TO NEW HARDWARE
=====================================
Building hardware is fun, but soner or later you come to the point where
you need software to do something meaningful.
So i started to look around. At work we had the BSD4.4 tapes and it included
a pmax port which was a good starting point. However BSD4.4 contain a few
licenced modules. To make this story short, NetBSD solved the issue.
A toolchain was needed to compile the code for my hardware and GNU was the
natural choise. Then i needed a porting plattform. So i had to dig the dungeon.
In my basement i had an old 68020 SYSVR3 un*x system which would do.
Next step was to create a cross development environment on the 68k system.
This was fairly easy, GCC runs on almost any system you might have.
I was now able to compile a NetBSD kernel from the pmax sources. After having
set up the include file hireachy the compile went through smoothly. Of course
i had no possibility to verify that it really worked. But as a starting point
for the rest of the work it could be considered as a milestone.
Then the time had come to start modifying and rewriting drivers for my own
system. I also had to create a bootstrap prom. Downloading and test took a
while because the only way i could download was via the serial port. Well,
after a few iterations (146 according to the version counter) the kernel was
working. At least until it tried mount the root disk.
Now the 'fun' part starts. Because i had no access to a BSD system i couldn't
create a file system on the hard disk. So i had to do it 'native'. The way
to do this was to hack "newfs" to run standalone with the bootprom hard disk
driver. Newfs was also hacked to create inodes for '/dev/console', '/dev/rz0a'
and "/dev/tz0". From this point i had a system which paniced because there
was no init process.
Next step was to create the rest of the system software. The tiresome process
of compiling all neccesary librarys was started. The libs was then installed
in the cross development system. It was now possible to cross compile the rest
of the system and create a distribution file system tree. Now how to download
the file system tree on the NetBSD disk? The file tree was "tared" out on a
streamer tape and i now had a 'system tape'. The problem to solve now was how
to get the tape down on the disk.
To load the tape to the NetBSD disk i needed a tar program that i could run
on the target system. To do this the kernel was hacked. The first hack was to
make the kernel mount the root disk r/w, open '/dev/tz0', copy the tape to a
file in the root directory, syncing the disks and then halt. What was there on
the tape then? Some of you might have guessed; a hacked :-) tar program. This
tar mounted the root disk, opened '/dev/console' and then started with options
'xvf /dev/tz0'. The kernel was hacked again to started this program instead
of init and downloaded. The 'distribution' tar tape was loaded, the hacked
kernel started and.... ta da da da, the tape was loaded to the hard disk.
Next step, reeboot with the downloaded kernel and viola, the system asked
for the shell to run!! I entered 'sh' and the '#' prompt showed up on the
terminal. Hurray!! It works, well almost. A few more things had to be fixed
in locore.s and the terminal driver.
From this point the rest of the system was crosscompiled and the last thing
crosscompiled was the complete GNU toolchain. Next step was to recreate the
whole system native and the when this was done, the plug was pulled on the
old 68k system.
Of course it requiered a number of tries before the download succeded. But
what was most amazing was that the system programs, sh, init, almost all of
the just worked! My only misstake was that the system i built was big-endian.
To all of you who made it to here, hope you enjoyed the reading.
I would really like to help you guys to get NetBSD for the MIPS boxes
housebroken. Most of my spare time right now is spent on porting it to
a R4400 board i got recently. The kernel will of course be different but
the rest of the system would pretty much be the same as for the pmax.
One suggestion that i have, based on my own experience, is that someone
creates a pmax/Ultrix GCC environment that people who compiles kernels
can use and put it up for ftp. You can then stop using Ultrix tools.
I will not take up more of your time right now, but if anyone has some
ideas let me know if and how i can be of any help.
Regards
Per
You can imagine all the list subscribers with their mouths wide open in awe.
I know I was when I first read that mail.
(Three days later, Jonathan Stone and Theo de Raadt had their
first
clash over a technical discussion about MIPS write buffers.
This would end up with Theo de Raadt expelled from the NetBSD project a few
months later.)
1995-1996, OpenBSD
In the meantime, Per Fogelström kept working on the R4400-based board he had
been referring to in his epic mail: an
Acer PICA.
When the OpenBSD project started for real in october 1995, Theo de Raadt, who
had also been able to get an Acer PICA, convinced Fogelström to share his code
with him, and that became the OpenBSD/pica port.
The PICA was a machine compatible with the
ARC
(Advanced Risc Computing) specification,
but there were a dozen or so ARC-compatible machines,
manufactured by DeskStation, MIPS, Olivetti and NEC, among others.
It made thus sense to support these compatible machines under a more
representative name, and in late august 1996, the OpenBSD/pica port was
superseded by a new version of its codebase, known as the
OpenBSD/arc port.
ARC-compatible MIPS systems were available for a few years and then, after
Microsoft announced Windows NT 4.0 would be the last version to support this
platform, disappeared completely. As a Unix workstations collector, one of my
(many) frustrations is that I have never seen any ARC machine on the second-hand
market in Europe. I really would like to eventually find one, but it looks like
these machines either suffer from electronic failures, or their owner does not
want to part with them, something I understand very well.
As summarized by Theo de Raadt some time in 2004 when
GXemul (still called
mips64emul at that time) grew ARC support and this was shortly discussed in the
OpenBSD private chat:
dead junk hardware. i had 3 in a row, and each of the boards died.
1996, Linux
Meanwhile, at SGI, there were a few Linux-enthusiasts who thought that it
would be nice to see Linux run on the low-end systems of the moment, especially
the Indy; and given how Indy and Indigo2 were close to each other, there was
hope of running on the Indigo2 as well eventually.
That group was led by Ariel Faigon, who first got a mailinglist created
to discuss that idea, and second, with the help of Larry McVoy who was
working at SGI at that time, convinced Dave Miller of linux-sparc fame to
spend a three-months internship at SGI to work on a Linux port.
(That mailinglist, initially hosted at SGI as linux@engr.sgi.com, would move
some years later and become the linux-mips mailing list, no longer affiliated to
any vendor in particular. The loss of the entire linux-mips site and content
due to a hardware failure a few years ago has been a tragic loss; thankfully,
some of its contents are still available on the
Wayback
Machine.)
Few people remember this, but Dave Miller was also an OpenBSD developer
during the first few months of the project, where he was working both on OpenBSD
and Linux sparc ports. Once Linux did run reliably enough on sparc, he stopped
contributing to OpenBSD, his last OpenBSD commit being on the 14th of january
1996.
To: linux list at sgi Subject: He he he Date: Fri, 15 Mar 1996 20:45:18 -0800 From: Larry McVoy I like news like this. By the way, we are really really close to signing David as a summer intern here. He's certainl'y a handful (the only person that I know that swears more than I do, and if you've hung out in B9, you know that's saying a lot :-) We're working on ways to channel all that energy and I think we have a plan. As soon as it is official, I'll post the details here. I think Ariel wants to have a Linux on SGI kickoff meeting soon - I hope folks are hip to that. ------- Forwarded Message Date: Fri, 15 Mar 1996 21:52:22 -0500 From: David S. Miller To: Larry McVoy Subject: The Ultra 176MHZ I'm not impressed with it's performance at all. Ho hum... maybe that will change when I get Linux running on it, perhaps Solaris is the problem here. Later, David S. Miller ------- End of Forwarded Message
Subject: Linux/MIPS port resources To: linux list at sgi Date: Fri, 19 Apr 1996 11:47:45 -0700 (PDT) I was thinking about what should we do to make David Miller come up to speed as fast as possible when he comes for the summer. After all, he may not be familiar with MIPS assembly, IRIX etc. So I set up a *preliminary* Web page with a list of resources for the Linux/MIPS port. Most of the pointers are from Bill Earl, thanks Bill. Please send me suggestions for improvement, what's missing, etc. This is just a quick first shot so you get the idea. The 6.2 freeware gcc is not yet configured to work with GNU-as (so it uses stabs and supports debugging) and our linker. I'll be working on this next week. We also need to make sure that all the equipment, the office, etc. is ready when David lands here. I assume someone is taking care of all this (?). And that someone with intimate knowledge of our low level stuff is really available to assist him on call. I'll leave it to Larry or Greg to announce the details on David Miller's accepting SGI's offer. p l e a s e :-) 6 weeks to go... -- Peace, Ariel
Date: Fri, 19 Apr 1996 14:52:28 -0700
From: Larry McVoy
To: linux list at sgi
Subject: Good news
Hi-
as some of you know, we've been negociating to get David Miller,
the Sparc/Linux guy, to come out and work on a MIPS/Linux port. He has
accepted, he starts on May 25th, and will be here until August 25th.
We had to do a lot of work with the laywers, but we have agreement that
all of the work that he does here will be
a) owned by SGI (we paid for it), and b) distributed under the
terms of the GPL.
No exceptions. SGI owns the code so we can choose to use anything that
turns out to be interesting inside IRIX without the constraints of the
GPL (you may or may not be aware that the owner of the code can choose
to distribute the code under multiple copyrights - so we can use stuff
in IRIX without "polluting" the IRIX kernel with the GPL).
I'd like to thank everyone that has been pulling for this, and especially
Greg Chesson who did the hard work of getting the contract hammered out
to the satisfaction of SGI & David.
We are currently in the process of figuring out what code we can use to
help with the port; there may be parts of the setup OS that are both
appropriate and useful.
Ariel and others are working to get a development machine set up in
the engr domain. It will be linux.engr.sgi.com, and should be up and
running on Monday or Tuesday.
I'll keep you posted on new news as it happens.
--lm
---
Larry McVoy lm@sgi.com http://reality.sgi.com/lm (415) 933-1804
Copyright 1996, all rights reserved. Microsoft Network is prohibited from
redistributing this work in any form, in whole or in part without license.
License to distribute this work is available to Microsoft at $500.
Transmission without permission constitutes an agreement to these terms.
I also still enjoy that particular mail:
Date: Mon, 22 Apr 1996 21:40:01 -0400
From: David S. Miller
To: Ariel Faigon
CC: linux list at sgi
Subject: Re: David Miller is on the list
From: Ariel Faigon
Date: Mon, 22 Apr 1996 18:16:30 -0700 (PDT)
Great, now you can tell the list what do you need :-)
Oh boy.
Seriously, we've been thinking about how we could make you most
productive and not waste your time when you arrive here.
Here's a recent posting of mine, so you get the idea just in case
Simon or Bob or Larry didn't tell you about all this yet...
Feel free to bombard us with requests/questions.
There are about 20 people on the linux list at SGI
(you can query majordomo@engr.sgi.com for the details)
Already did that an hour ago ;)
(note: I looked back over this mail after composing it and I want to
warn people who are not familiar with me yet that I am very
sarcastic and am full of ridicule even when discussing
important topics. Please don't take it that I lack tact
or am not being serious, because that simply isn't the case.)
Here is what I need:
The following utilities I need for development.
1) CVS/RCS, latest on prep.ai.mit.edu is fine
2) Emacs-19.31 (rms should release within 2 weeks)
3) All GNU smidgen-type utilies as the default binaries
(this include fileutils/sh-utils/sharutils/diffutils/
findutils/...)
Actually, Let me just stop short and say, if there is a
source tarball for it on prep.ai.mit.edu:/pub/gnu I would
like the latest installed on the machine I develop on.
4) xfishtank (don't laugh)
5) fvwm
6) teco (Must support full teco command set as described
in original DEC manuals! TECO is _the_ renaissance editor!)
The following would be nice, but if it will give people
bladder problems to do these then don't go out of your
way:
1) MIPS 4[40]00 manual is some online format (not postscript,
something I can cut and paste out of an emacs buffer etc.
so maybe info or pure ascii text would be fine, I could
care less about the formatting, I just want the words
there)
2) Docs on the ethernet/scsi interfaces and I/O bus
architecture for the first machine I will be getting
this to work on, again text/info format would be nice.
Of course I will probably just stuff in the ready
drivers you might be getting to me into Linux but I want
to write my own from scratch in the near future after
that.
3) I know as much as a bum on the street about SGI machines
and the various lines, a nice "roadmap to sgi workstations
and servers, plus the hardware gook thats inside" type
thing would be very useful to me.
I will feel more comfortable if:
1) I became very familiar with who the heavy low level MIPS
assembly level hackers are who I will be dealing with while
I am there. Please tell me who they are, introduce, make
us say hello to each other, you get the idea.
2) I know the policy on loud music in the office I'll be in
;-)
I've thought it over and to me the best plan for things this summer to
me is:
a) R4400 32-bit "proof of concept, yeah we can pull it off"
port happens first, side effect is that I become intimate
enough with the chip that I can do things more efficiently.
b) From here we look into the 64-bit stuff and whether that is
is even desirable on 64-bit. (this would be my first
64-bit port outside of my initial UltraSparc hacks)
c) Also think about the work needed to turn that code into
r3000 friendly code. Should not be too much as I've done
the "write it on recent architecture design then backport
it to older design which had some limitations" already and
this didn't end up being so bad.
Expect more as I think it up... this should keep you guys busy for
now.
(Any dead-head tape traders at SGI engineering? Just wondering, may
want to start talking to them now ;-)
Later,
David S. Miller
…where we get a glimpse of the work setup of Dave Miller, including the use
of
xfishtank
(thus of a 8-bit frame buffer, as xfishtank did
not yet support true colour visuals), and the use of both
Emacs and
TECO.
Also, in a later mail, he mentioned it was
fvwm 1.24l he was
running, not the 2.x versions.
By the end of his intership, Linux was running on the Indy, with Ethernet and
SCSI support, and, if I remember correctly, some minimal glass console support
in addition to a working serial console. Ralf Bächle spent some time
after that to merge Dave Miller’s tree into the current linux-mips tree, as his
porting effort had been based off a different Linux kernel version
(2.0.14, while the current linux-mips kernel was 2.1.21).
Bringing in the sgi-specific parts, which were new, was the easy task, while
merging the useful fixes he had made to the common MIPS code required more
attention.
fall 1996 status
Note: there will be several status tables across the years. The IPxx
notation below corresponds to a given hardware design, with IP standing
for Inhouse Processor, to differentiate them from the very first SGI
workstations (Iris 1000) which were based upon a licensed third-party design
similar to the Sun-1; as you will see later in this narration, these IP
values appear often in developer communication as a way to precisely identify a
given piece of hardware. Plus it’s shorter, written that way!
| SGI model | common name | Linux |
| IP24 | Indy | kernel no bootloader no distribution XL (newport) graphics only no X server |
1997-1998, Linux
Ariel Faigon had hoped that Dave Miller would return to SGI for another
internship the next year, Miller had other plans and did not resume working on
SGI support for Linux. This task was taken over by Mike Shaver and
Miguel de Icaza, in addition to Ralf Bächle.
With the help of Alex de Vries, who would later initiate the PA-RISC
porting effort, they eventually managed to build a variant of the RedHat Linux
distribution for the Indy, called HardHat.
1997-1998, OpenBSD
Among all the MIPS hardware he had collected, Per Fogelström had an SGI Indy
machine. These systems use a firmware interface which is close to ARC, with
minor differences. It was reasonable to expect the existing OpenBSD/arc code to
work on this machine, with only minor changes.
And indeed, it did. Near the end of august 1997, Per Fogelström had the
beginning of a port, running diskless with a newport frame buffer console, and
rough edges which would disappear once people would spend enough time fixing
them.
(That codebase also hardcoded the Ethernet address of the onboard interface
to 08:00:69:08:9c:54; if you ever encounter an Indy machine with that
Ethernet address, you’ll know that it used be Per’s and that it is a small part
of OpenBSD history.)
Yet, for reasons unknown to me, this work was not commited to the OpenBSD
tree. In february 1998, Michael Shalayeff got access to a few SGI
O2
systems which were no longer in use at his dayjob, and asked Fogelström for
information about this hardware, but Fogelström didn’t know much about the
O2.
Shalayeff nevertheless asked him to share his work-in-progress OpenBSD/sgi code
with him, something Fogelström eventually did in march. Despite keeping telling
that he would commit that code soon, still nothing happened. Then Shalayeff got
busy with his own
PA-RISC porting work and gave
up on SGI, or at least didn’t find time to tinker with it anymore.
(It was really frustrating for me to discover this many years later, as I
strongly believe that, if there had been a visible effort to make OpenBSD run
on the Indy in 1997, there would probably have been much more interest, and
more other SGI systems would have been supported earlier.)
fall 1998 status
| SGI model | common name | Linux | OpenBSD |
| IP22 | Indigo2 | kernel no bootloader XL (newport) graphics only no X server |
|
| IP24 | Indy | kernel no bootloader XL (newport) graphics only no X server |
kernel no bootloader diskless XL (newport) graphics only no X server not public |
[go back to the previous part][skip
to the next part]
2000, Linux
Linux started to get working Origin 200 support in january. It took a few months
to work stably, and multiprocessor support only landed in april, and wasn’t
stable until june.
That code was heavily relying upon knowledge gathered from IRIX header files, to
the point some mistakes were made when “creating” Linux header files.
Date: Mon, 15 May 2000 19:38:28 +0000
From: Ralf Baechle
To: linux-cvs@oss.sgi.com
Subject: CVS Update@oss.sgi.com: linux
CVSROOT: /home/pub/cvs
Module name: linux
Changes by: ralf@oss.sgi.com 00/05/15 12:38:28
Modified files:
include/asm-mips64/sn: intr.h intr_public.h nmi.h
Log message:
Fix copyright message. Linus flamed me on these messages so please
make sure for the future that the (C) notice is ok. ``unpublished
proprietary information of Silicon Graphics'' isn't good ...
There was not much communication about that support, however, apart from
an article on the popular (back then) news site
slashdot, which I couldn’t find
anywhere in its archives anymore.
2000, NetBSD
In the spring of 2000, NetBSD developer Soren Jorvang started to work on
a port to the SGI O2, using information from the IRIX system headers as the only
documentation, and the existing NetBSD MIPS codebase as a solid fundation.
His work was first commited to the NetBSD tree on june 14th, with a first
complete NetBSD snapshot available on the 29th. However, possibly due to bugs in
the generic MIPS code in NetBSD, this code was not running as stably as
expected, and one of the first hurried fixes was to disable the second-level
cache of the R5000-family processors (R10000-family O2 were not supported yet.)
The O2 platform is a non-coherent architecture, when it comes
to its cache memory behaviour.
This means that there are no electronics in charge
of making sure that every device in the machine sees the same memory contents,
at any time. This matters for devices able to perform
Direct Memory
Access, such as most I/O controllers (on the O2, the SCSI
controller, the Ethernet controller, the Audio/Video board, etc.)
Non-coherent systems are frequently encountered in legacy workstations (and
even to this day in small computing devices, such as Raspberry Pi or StarFive
boards), and all operating systems running on such hardware will perform
explicit cache flushes and/or invalidations, in order to make sure that
no cache has, even partial, contents of the memory which is about to be
modified by a DMA transfer.
But some O2 add extra difficulty to that problem: the R10000
processor.
The R10000 processor performs
speculative
execution in order to have better performance: when a branch point
is reached, it tries to follow both “branch taken” and “branch not taken”
instruction flows, so that, at the time it is known whether the branch is
taken or not, it is not slowed down. Of course, the other half of the work,
for the wrong outcome of the branch, gets discarded.
In an ideal world, discarding the wrong set of instructions would be completely
transparent. But it is not – there are subtle, minor side-effects of
the speculative execution which can’t be undone (well, they could, but that
would require a lot more machinery in the processor, and it was deemed
acceptable to let these side-effects exist.)
This side effect is that, if a load instruction is speculatively processed,
the processor will perform the memory load. If the data is not found in cache,
this will cause it to be fetched from memory, and put in cache. If a store
instruction is speculatively processed, the same thing may happen, with the
cache line being marked as dirty (modified), so that it will get written back
to memory when it needs to be evicted from the cache.
So if a speculative store is processed using a memory address which will be
written to by a DMA operation, there is a risk that the processor’s view of
the memory area will be wrong, and there is also a risk that the processor
will write back outdated data from its cache to that area.
In order to prevent these unwanted side effects, some processor instructions
can act as speculation barriers, blocking further speculative execution
until the instruction completes; and there are also several code generation
tricks, documented in the R10000 processor manual, to reduce the risk of
unwanted side effects.
Speculative execution has no ill effects when the processor is running userland
code, because accesses to addresses which would cause a MMU fault (because the
access would be made to an address which computation would involve a register
value, and the register value is nonsensical in the stream of instructions which
will be discarded, for example.) But the kernel has access to the complete
physical memory through the XKPHYS area of the 64-bit virtual address
space, and these addresses do not need to be translated by the MMU.
Because of this feature, the risk of speculative execution causing these side
effects was considered nonzero by the O2 designers, and they
implemented an extra defense against them in IRIX:
- The O2 memory controller has a configurable limit, above
which speculative load and stores will be forcibly aborted. - The IRIX kernel programs this limit value so that the kernel image
(including kernel data) is entirely contained below the limit,
and makes sure to allocate its DMA buffers above the limit.
Because of the existence of that feature, it was expected that support for
the R10000-based O2 in Linux and *BSD (which could use the same
workaround as IRIX, but don’t actually do anything) could not be easily
achieved.
2001, NetBSD
The sad situation of the O2 support did not prevent people from tinkering with
the idea of running NetBSD on their SGI machine, even if it was not an
O2.
In late april, Rafal K. Boni had his Indigo2
run
multiuser.
Subject: Indigo2 boots to multiuser!
To: port-sgimips
From: Rafal Boni
Date: 04/27/2001 10:51:43
Folks:
I've gotten my Indigo2 booting well enough that it gets to multi-
user mode, and seems to be able to run most things fairly well.
I haven't tried anything like building a source tree yet, but
we'll get there at some point.
There's still loads of work to be done (notice we have no RTC
driver, hence the box things it's at the start of the unix
epoch, there is no SCSI driver so this is all over NFS, the
ZS driver isn't in good enough shape to boot single-user yet
so this is using the ARCS console...) but thought this was
exciting enough to send out some mail.
[...]
His work was commited to the NetBSD source tree by Jason Thorpe on may
11th.
Although the O2 support was still in bad shape, this was an important milestone
as it allowed the SGI-specific code to be better split into generic “a machine
running a MIPS processor on SGI’s variant of the ARCBіos firmware” code, and
per-particular-model code.
An SCSI controller driver for the Indy and Indigo2 was added in
august by the late Wayne Knowles, who significantly improved it in
november.
The last important code piece, a standalone bootloader, allowing the NetBSD
kernel to be booted from disk instead of accross the network,
was pieced by Michael Hitch in november 2001.
2001, Linux
There had been patches floating around to let Linux run on the O2 systems, after
NetBSD showed the way. But none of them would be seriously crafted, often
breaking support for Indy while adding O2 support, because the people working on
O2 support did not necessarily also have an Indy or Indigo2 machine,
or were not being careful enough making their changes.
This changed in december 2001 when Vivien Chappelier, then a student at
the
École
Nationale Supérieure des Télécommunications in France, started to spend
quality time with his O2, and submitted patches which were not always correct,
yet were of a much higher quality than previous efforts. This allowed him to
receive valuable feedback, improve his code, and get his patches eventually
accepted to the linux-mips tree.
fall 2001 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP22 | Indigo2 | kernel no bootloader XL (newport) graphics only |
complete distribution no graphics |
|
| IP24 | Indy | kernel no bootloader XL (newport) graphics only |
complete distribution no graphics |
|
| IP27 | Origin 200, Origin 2000 | kernel no bootloader |
||
| IP32 | O2 | not-yet-integrated kernel patches no R10000 support no bootloader no onboard Ethernet no graphics |
complete distribution no R10000 support no graphics no onboard Ethernet kernel does not run stably |
2002, Linux
Vivien Chappelier did not remain inactive, and contributed support for the
O2
on-board Ethernet interface in july, and a frame buffer driver in december.
Late november, the Origin 200 and Origin 2000 code, which had be rotting and
was no longer in working condition, got repaired and improved by Ralf Bächle,
restoring proper support for these systems.
2002, NetBSD
O2 support was reported as
“very
broken”
at the beginning of the year.
Subject: Re: O2 doesn't boot To: port-sgimips From: Christopher SEKIYA Date: 01/03/2002 20:35:30 On Thu, Jan 03, 2002 at 06:11:23PM +0900, Toru Nishimura wrote: > It should not be a hard job to tailour NetBSD/sgimips for RM5200, I think. O2 support is very broken: * The walk-the-ARC-tree-for-cache-detection code does not work properly, * watchdog is turned on too early, breaking console settings slower than 38400, * PCI code handles memory space mapping but not IO space mapping, breaking most ethernet cards, * interrupt masks are incorrect, * interrupt handler is very spooky and frequently deadlocks the machine. There's more that's wrong, but that's an executive overview. Adding the proper logic for a 5200 won't result in a working machine. I've fixed #2 and mostly fixed #4 and #5 from the above list, have worked around #1 and found a card that will mitigate #3, but my O2 will still fall over three times out of five once interrupts are turned on. Is anyone else working on fixing the O2 support? I'll send what I have to interested parties. -- Chris
However, by the very end of the year, Christopher Sekiya had reached a
state where his O2 was
running
stably again.
From: Christopher SEKIYA To: port-sgimips Subject: New IP32 kernel/patchset Date: 12/29/2002 11:04:03 Ladies and gentlemen, I've put diffs against today's -current and a test kernel for the IP32 at http://www.rezrov.net/software/netbsd/ip32-diff-20021229 and http://www.rezrov.net/software/netbsd/netbsd-ip32-20021228, respectively. Apart from cosmetic changes to facilitate repository commits, the following changes have been made since Rafal's last snapshot: * the timer is properly calibrated. The system should no longer consistently lose time. * ahc is recognized as a boot device. * the crime interrupt handler more closely resembles a "real" interrupt handler. In conjunction with boot.ip32, I've succeeded in booting my IP32 from disk straight to multiuser. The system appears to be stable. Share and enjoy, -- Chris
2002, OpenBSD
After almost three years of inactivity, Per Fogelström found time and motivation
to resume working on the OpenBSD codebase, firstly for his own company
(Opsycon AB) and its PMON2000 extensible firmware product. With the availability
of embedded boards with high-end MIPS chips, there was an interest for a
64-bit MIPS version of PMON2000… and also for a 64-bit MIPS version of
OpenBSD.
But 64-bit MIPS support in the GNU toolchain (gcc and binutils) was still in its
early beginnings, with many 64-bit specific bugs in gcc (while 32-bit MIPS code
generation was quite reliable) and 64-bit MIPS ELF targets in binutils were
new and not well-tested yet.
Over the last week of february 2002, on the private OpenBSD developer chatroom,
he mentioned his current progress multiple times.
well, i'm trying to move the mips kernel to 64 bit, not with full 64 bit addressing yet, but run into issues here and there. [...] hi what version of binutils is in the tree right now? 2.10.1 is this what is going to be in [OpenBSD] 3.1 or will an upgrade be done before that? mips have problems in 2.10 and needs 2.11 thats why i'm wondering. [...] morning what a dull day... hmmm. 5.3 hours for a make build under an Ultra 2, at 400MHz. takes 4,4 hours on a 400Mhz mips machine. go get sgi's a port is coming! :-)
And here we go, the idea of an OpenBSD/sgi port is coming back.
The day after:
my 64bit mips kernel is now starting to run init. looks good so far but still things that needs to be fixed. looking forward to that O2.
And the next day:
heh, OpenBSD/mips64 now have shell prompt! :-) sweet and mips32? :-) don't tell theo but it runs in o32 emul mode. :-) mips32 per se is already working fine. i'm trying to keep the kernel code as generic as possible so if someone insist on making an OpenBSD/mips32 it should work. more or less a matter of compilation options. now i can't think of any reason why to make a mips32 kernel unless someone wants to start fighting old R3000 hw again... no don't look at me. a mips64 kernel is only about 10% larger than a mips32 kernel.
Two weeks later:
nice to have some good progress now and then. mips64 is starting to work really well. still much work but it looks really good so far. pefo, do you plan on merging mips anytime soon? as soon as i have all 64 bit pieces going. actually theo says he want it clean and fixed before importing it. there are a few issues regarding endianness etc that has to be dealt with. i'd be glad to help i'm currently playing with gcc 3.0.4 and binutils 2.12 which seems to work pretty well so far. i know. binutils 2.11 cannot handle mips64, right? 2.11 has a lot of problems. 2.11.94 which is close to 2.12 handles o32, o64 but not yet n32 and 64. 2.11 can do o64. i've been compiling the kernel with that until now. that is the 64 bit kernel. i've never got even o64 working with 2.11 here. the IP30 prom can only load a 64bit kernel, IP32 seems more flexible. " the IP30 prom can only load a 64bit kernel" do you by that mean it can only load ELF64? yes i see, thats a different thing. 2.11 will not do that. it will only work ok when doing ELF32 files. what kind of SGI machines do you have? Indy, Challenge S, Indigo2, O2 and single-CPU Octane cool, looks like you are going to get busy! :-) that is, if i can get my hands on your code =) i'm curious to see how you're handling caches well, caches seems to be one of the hardest things to grasp... L2 caches are a pain well, as long as one remembers to deal wit orphans it's no problem. then if you define a set of cache operations coupled to the action instead of encapsulating the various cache commands in functions it's really easy to deal with it. but people always aproach the problem from the wrong direction. then not understanding the behaviour of non-snooping caches and think that busdma will be the universal solution is a *big* mistake... put these things together with some aliasing, and voila! you have stability problems.
One week later, a short status update:
well, i have a working (to some extent) kernel and have made a simple o32 emulation to allow the older o32 binaries to run. not everything will work but i wanted something to help me bootstrap the 64 bit world. a couple of strange things happen which i need to track down to either compiler/toolchain problems or my code. how many boxes do you run on? anything common for our developers to get? three. haven't started sgi yet. wcobb is hot on it though :-) have an O2 standing beside me that i'm going to play with soon Good. there are small fast sgi machines, right? have spent quite some time on getting the tools to do the right thing and have 80% of 64 bit userland built -static. shared stuff later if o64 hold water. all new code i create under mips64 is supposed to work in ISA2-ISA4 mode, eg both 32 and 64 bit mode. but i'm not going to do anything with ISA1, eg R3000 now. it's to painful...
A few bugfixes later, at the end of march, our hopes of getting a 64-bit sgi
port were high.
the SGI porting work will really take off now. i have an Indy, and an Octane on its way here from US. what is the SGI machine status around here, what do we got? Indigo2 i know, what else? yeah, i have an O2 also.
Two weeks later in april, the same optimistism:
morning. the probability for [OpenBSD] 3.2 to have a SGI port is becoming higher and higher. good. what hardware is working? o2 is my first target. will attack that hard next week. if it is getting close, pefo, it is going to become time for you to hook a few developers up with the right hardware soon. like miod, me, and a willing ports person, probably one of brad (toronto) or naddy (germany). so that we can do our end of the cleaning wcobb is looking at the octane. i will go back to the indy after i have the o2 going. well you will need to ensure we get hardware and please, small hardware? :-) small + fast + cheap (the latter is not as important) so if things work ok at least indy, o2 and octane will be supported as a start. great. you can get an indy for $100 SGI hardware is relatively cheap these days. what is the fastest PC-size sgi? I have no idea the o2 with a R12000. still rather expensive. you can the a 200Mhz R5K o2 for about $250-500. sometimes they go really cheap on ebay. and how fast is it Yeah, I would think a cheap octane would be the way to go octanes is hard to get under $500-600. But that is not too expensive well, for the first few, we will just have to ask. fast is relative. but if we look at tree build performance an r5k/200 will probably build the tree in 6-7 hrs. now it feels that gcc 3.x is becoming even slower though... well, that isn't too horrible. we do not have any middle of the line machines at the moment < 3 hour and > 20 hours. if get the O2 port up and running it is possible that we can get a few donated from where i got some of the extra stuff for the octane i bought.
And then there were no further signs of life from Per…
Late may:
what's up with the sgi stuff? haven't seen pefo here in a while Who knows.
Fogelström appeared again in december, with an important question.
Date: Mon, 30 Dec 2002 17:58:30 -0800 From: Per Fogelström To: private OpenBSD mailinglist Subject: SGI system type poll Hi, I've now started to code the SGIMIPS kernel and thought i should check what type of systems you guys have. My first target will be the O2 since that is what i have around and what i consider reasonable. The box that sits here is a R5000 so that will be what is going to work first. The other box i have is a R10000 but it's on loan right now but if there is people with O2 R10K's it will be the next target. I have also started to search for O2 frambuffer info and if anyone have any i would appreciate that. Since the O2 is a shared memory buffer design it would probably be fairly easy to get a dumb framebuffer X port working when the details are uncovered. Once the kernel is operational i think there is little work left to have almost everything working. I currently have 1600+ packages compiled although not everything is tested! :-) At least KDE and some other stuff is working fine on my Galileo/Marvell Discovery system as i'm typing this using KMail... So stay tuned! :-) Per
There were not many reactions. I said I was more interested in Indy and
Indigo2
support at the moment since these were the systems I had, but wouldn’t mind if
other systems were supported first. It looked like the general attitude was
“wait and see”.
fall 2002 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP22 | Indigo2 | kernel no bootloader XL (newport) graphics only |
complete distribution no X server |
|
| IP24 | Indy | kernel no bootloader XL (newport) graphics only |
complete distribution no X server |
|
| IP27 | Origin 200, Origin 2000 | kernel no bootloader |
||
| IP32 | O2 | not-yet-integrated kernel patches no R10000 support no bootloader |
complete distribution no R10000 support no graphics no onboard Ethernet kernel does not run stably |
likely some private tinkering no R10000 support status unknown |
2003, Linux
Not much happened for Linux on SGI that year.
Guido Guenther wrote a standalone boot loader,
arcboot,
allowing Indy and O2 to finally boot Linux from disk.
In a message to the linux-mips mailing list in november, Ralf Bächle
acknowledged that Linux on the O2 was not really ready for end-users.
Date: Wed, 12 Nov 2003 14:53:33 +0100 From: Ralf Baechle To: BETTLER Emmanuel Cc: linux-mips Subject: Re: O2 and Linux boot problem [...] Linux for O2 is still in it's very early stages, don't expect much success with it for the moment. Hackers welcome :) Ralf
2003, NetBSD
On the older side of the SGI hardware spectrum, Steve Rumble made NetBSD
run
on the R3000-based Indigo (IP12, the first generation of Indigo systems)
in april and shared his work-in-progress code, which would however not get
commited until one year later.
On the more modern side of the spectrum, there was not much visible progress
this year, until the last quarter of the year, where O2 support was repaired
in october thanks to the efforts of Christopher Sekiya, and R10000 processor
support was added shortly afterwards.
On december 15th, glass console for Indy systems with the “newport” (XL) frame
buffers
was
added.
And to end the year on a good note, Christopher Sekiya added support for the
R4000-based Indigo (IP20, the second generation of Indigo systems) on
december
31st.
2003, OpenBSD
Fogelström came back early january, to complain about the state of OpenBSD’s
driver for the Adaptec AIC-7880 SCSI controller found onboard the SGI
O2, which
code was not endian-neutral at that time.
aic7xxx.c is severly broken. i found a lot of BE issues. swapping of already swapped adresses, faulty busdma_syncs... looks like i have a lot of work to do here... :( NetBSD don't use our code. it's not based on the same FreeBSD version even. ahc0: PCI error Interrupt at seqaddr = 0x7f <-- from OpenBSD on SGI O2. not much choice. i need to get this driver working. ahc? why ahc? aic7880 <- builtin on the O2. two of them. the free pci slot is needed for an ethernet board until the builtin ethernet is figured out. wow. what is the builtin ethernet? something 'homemade'. sits in a big asic. seeq? hmm, i think it's thunderlan (at least that what i had on my r5k) sgi owns seeq, if i recall. that could be found out while looking at the published .h files. have to go. cinema time. btw, whoever wants to play with ethernet is welcome! :) well i do not have mine anyymore (; at least the kernel is coming alive. another week or so i would guess...
It only took one day of work for the situation to improve, as reported on the
next day:
sd0 at scsibus0 targ 1 lun 0: SCSI2 0/direct fixed :) from the O2 heh, cool! so you have ahc BE clean now? If you do, I would like to see those diffs. Whoa, BE clean ahc? That would ROCK. yes, it was (so far) two places of swaps and one missing dmasync. gimme wonder if that could help with some of the general ahc probs... not on i386... no dmasync's necessary on that arch. dmasync, no. dmasync and BE will help sparcs with PCI though.
And one day later:
showtime: > boot -f bootp():bsd.sgi Setting $netaddr to 192.168.16.70 (from server mammoth) Obtaining bsd.sgi from server mammoth 2140912+246880 entry: 0x80100080 Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2002 OpenBSD. All rights reserved. http://www.OpenBSD.org OpenBSD 3.2-current (GENERIC) #97: Thu Jan 9 04:21:38 CET 2003 pefo@mammoth:/archive/OpenBSD/3.2/src/sys/arch/sgimips/compile/GENERIC real mem = 67108864 avail mem = 58425344 using 819 buffers containing 3354624 bytes of memory mainbus0 (root) cpu0 at mainbus0: MIPS R5000 CPU Rev. 2.1 with MIPS R5000 based FPC Rev. 1.0 L1 Cache: I size 32kb(32 line), D size 32kb(32 line), two way. macebus0 at mainbus0 macepcibr0 at macebus0: chip revision 1, host system O2. pci0 at macepcibr0 bus 0 ahc0 at pci0 dev 1 function 0 vendor 0x9004 product 0x8078 rev 0x00: irq 1 ahc0: aic7880: Wide Channel A, SCSI Id=0, 16/255 SCBs scsibus0 at ahc0: 16 targets sd0 at scsibus0 targ 1 lun 0: SCSI2 0/direct fixed sd0: 2049MB, 8188 cyl, 3 head, 170 sec, 512 bytes/sec, 4197405 sec total cd0 at scsibus0 targ 4 lun 0: SCSI2 5/cdrom removable ahc1 at pci0 dev 2 function 0 vendor 0x9004 product 0x8078 rev 0x00: irq 2 ahc1: aic7880: Wide Channel A, SCSI Id=0, 16/255 SCBs scsibus1 at ahc1: 16 targets fxp0 at pci0 dev 3 function 0 vendor 0x8086 product 0x1229 rev 0x08: irq 4, address 00:90:27:a5:47:58 inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4 com0 at macebus0: ns16550a, 16 byte fifo com0: console com1 at macebus0: ns16550a, 16 byte fifo boot device: lookup 'unknown' failed. root device : sweet =P next thing is fixing the interrupts and i think this little toaster will run :) the oxygen toaster ? yup oh hmm, it has fxp onboard, i bet i had thunderlan no, it's in the pci slot. the onboard ethernet is still unknown... ah it's not on pci ? the fxp is a quickstart solution until the built in ethernet is working no, it's not on pci. it's in the same asic as the pci bridge but not a generic pci device.
There are a few interestings things to observe in the above boot sequence:
- the name of the machine the kernel was built on is mammoth, which
proves that, unlike what we’ve been told, mammoths were still alive in the 21st
century
(although with only 64MB of memory and no L2 cache, it was obviously a small,
endangered, specimen.) - the port name was still sgimips at that time, similar to the NetBSD
port name.
And of course, Fogelström disappeared again, until early september:
[=Sign-on=] pefo (pefo@[XXX.XXX.XX.XX]) entered groupholy shit, a swedish developer we've not seen a swedish developer in weeks yeah... hello pefo hi miod probably because it's their 6 month night now so is sgi ever going to happen? hi pefo summer have ended and we are crawling back to our caves ;-) hi todd I've been bugging pefo to get his sgi's back online. rumor has it one is .. OK, a mips machine is available for remote access for those who wants to hack mips code. It runs 3.3 right now and have my mips/sgi sources on line. do you want to incorporate things into the tree after unlock? sgi/O2 stuff can probably go in. i'm rearranging the tree per your request first. the only thing i want to do before going public with things is to move over to n32/n64 abis as default. n32/n64? err, isn't n32/n64 the abi horribly poorly supported by binutils? afaik they have decent o32 code, but nothing for n32 do you mean i32lp64? i do not know this other notation the new ABIs. Have better call methods (faster), uses 8 arg regs instead of 4 and have less overhead. oh, it is an abi, is it on top of i32lp64? not any longer. yes. n32 is i32 n64 is lp64 i'd say no because I only consider 32bit systems here (-: pefo, you mean you have written n32/n64 code for binutils? no, not me but redhat. oh. I was not aware of this. that's excellent news. i have the code and i think it is in the cvs now also. there are some specific tweaks for the RM9000 which i'm not sure about but that doesn't matter yet. this is still the newer SGI stuff, my Indigo2 is still SOL, right? nick, you can enjoy netbsd and serial console only on it (-: and no L2 cache (unless they fixed this recently) oh, and those two wonderful, SGI-only 20" monitors. *sigh* Indigo2 may come later. We have to start somewhere and O2 is a good place. It's a decent machine both in speed availability and price. [...] pefo, you got propolice working? too bad mips is totally incapable of W^X yes, its no news. The R10K is a little "special". There may be a couple of rare RM52xx and RM7000 around as well. Those are OK. i'd expect pp to misbehave on mips, if only because mips cpus smell bugs. Even though they're a dream to program for, from an userland pov... but then i also expected it to misbehave on m88k and it did not. Yes i'm running 3.3 with stack protector on. no problems seen so far. pefo, contact etoh to see if he wants to do any tests make build goes thru with no problems. ok
difficult to understand.
The interesting part of this conversation is the discussion about the
ABIs.
At that time, there had been already four levels of instruction sets for MIPS
processors, each level being a superset of the previous one.
- MIPS1 is the original MIPS instruction set, supported by the R2000 and R3000
families. - MIPS2, introduced with the (obscure, short-lived) R6000, adds “load linked”
and “store conditional” instructions allowing “compare-and-swap” atomic
sequences, as well as “branch likely” flavours of branch instructions. - MIPS3 adds 64-bit support, introduced with the R4000 family.
- MIPS4 adds a few more instructions (conditional moves and some more
floating-point instruction flavours). It was first introduced
with the R8000 processor, then also supported on the R10000 family and its
variants (R12000, R14000, R16000.)
Since OpenBSD/sgi was to be a 64-bit port, it would target MIPS3 and MIPS4
processors only – from the R4000 up.
In the early MIPS days, when it was still a 32-bit processor, the original ABI
for MIPS code allowed for up to 4 function call arguments to be passed in
registers, any further argument being passed on the stack. MIPS processors
having 32 general-purpose registers, this was a horrible mistake, and having
allowed more registers for parameter passing would have been better, but it was
too late to fix that.
But when 64-bit support was introduced, MIPS engineers seized the opportunity
to fix that, and designed a better calling convention for 64-bit
code, using up to 8 registers for parameter passing.
This new calling convention was called n64 (n for new),
and in retrospect, the old 32-bit calling convention was called o32
(o for old).
Then, users of 32-bit code realized it would be a good idea to also use 8
registers for parameter passing, and their adaptation of n64 to 32-bit
code was called n32.
Then, because “why not”, gcc added an o64 calling convention, which is
the equivalent of o32 for 64-bit code, and noone sane should use it
unless forced to.
For the OpenBSD port, it was clear n64 was the way to go, but this
required reliable code from binutils and gcc, and whether this code was reliable
was yet to be discovered.
fall 2003 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP12 | Indigo (R3000) | complete distribution no graphics |
||
| IP20 | Indigo (R4000) | complete distribution no graphics |
||
| IP22 | Indigo2 | complete distribution XL (newport) graphics only |
complete distribution no graphics |
|
| IP24 | Indy | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only, no X server |
|
| IP27 | Origin 200, Origin 2000 | complete distribution | ||
| IP32 | O2 | not-yet-integrated kernel patches no R10000 support |
complete distribution no graphics no onboard Ethernet |
kernel no bootloader no R10000 support no graphics no onboard Ethernet not public yet |
2004, Linux
In april, Stanislaw Skowronek appeared on the linux-mips list with the
first bits of Octane support.
Date: Mon, 12 Apr 2004 22:51:13 +0200 (MET DST) From: Stanislaw Skowronek To: linux-mips Subject: Work on IP30 My Octane got as far as 'Calibrating delay loop... ' now. It works with ARCS graphical console, which is a Good Thing. Memory identification is correct. Caches work OK. I'm going to do the IRQs tomorrow, but I foresee it will be really hard as there is no documentation for the HEART. Well, we shall experiment. Fortunately, the Octane has one really nice feature: the power button is hooked up to an interrupt source. It may prove quite useful for debugging. Interesting note: the ARCS console breaks when I change KSEG0 cache coherence in the CP0_CONFIG register (in c-r4k.c). Of course, it breaks sooner or later, not exactly afterwards, unless I flush cache exactly after changing; it breaks immediately then. I don't give a hoot, because IP30 has almost no RAM in KSEG0 and the kernel runs in XKPHYS, always cached copy-on-write. But those of you with another machines might be interested. Stanislaw Skowronek
A few days later, he had made progress:
Date: Fri, 16 Apr 2004 23:13:29 +0200 (MET DST) From: Stanislaw Skowronek To: linux-mips Subject: IP30 goes relatively far now Hello, I'm currently doing a reverse-engineered IP30 port of Linux-MIPS. Currently I'm using 2.6.1 as my base. I don't know if it's been already fixed in >2.6.1, but in genex.S there should be a 'nop' between 'jal do_\handler' and 'ret_from_exception'. The symptom is a hang on 'Checking for the daddi bug...'. Somebody apparently got used to '.set reorder' :P Well, now the kernel crashes a bit later. Actually, it gets to 'mice: PS/2 mouse device common for all mice' and then gets an Instruction bus error. I will look into this. Currently the kernel supports only MGRAS graphics (SI, SSI, MXI, SE, SSE, MXE) and uniprocessor. I don't have a SMP machine here, but I guess it would not be particularly hard to do. The ODYSSEY (VPro) would be a bit harder, as its architecture is vastly different from the MGRAS. Anyone interested may send me a VPro6 ;) When I get to 'cannot mount root', I will release the kernel patch. Yours, Stanislaw Skowronek
On the 25th, as promised, he shared his code.
Date: Sun, 25 Apr 2004 19:18:16 +0200 (MET DST)
From: Stanislaw Skowronek
To: linux-mips
Subject: Work on IP30
Hello!
I have placed a alpha version of my patch for Octane at:
http://www.et.put.poznan.pl/~sskowron/ip30/
Things currently unsupported:
* SMP will not work,
* qLogic SCSI driver oopses at the start,
* keyboard is not written yet,
* VPro (Odyssey) graphics,
* ARCS console,
* serial console,
* PCI card cage (trivial, but I can't test it as I don't have one)
Things not yet tested:
* new Octanes and Octane2s,
* user mode (yes! I don't have yet a MIPS glibc to cross-compile for
user mode),
* almost everything
Stanislaw Skowronek
The next month, he was able to run userland, albeit diskless, as there were
still problems with the SCSI driver.
Date: Mon, 10 May 2004 22:54:03 +0200 (MET DST) From: Stanislaw Skowronek To: linux-mips Subject: Octane - first light! Well, we've got usermode now. There is no keyboard driver so I can't comment on interactive work (and I am too lazy to set up a complete telnet server :), but all interesting scripts I gave it worked. Network is OK, the QLogic SCSI driver does not work for some reason or another (it can't communicate with hardware - why?), the console driver is nice and seems to work rock-solid, high IRQ load (ping -f) does not break anything. I found another 32-bit infelicity in kernel/scall64-o32.S in stackargs: using subu instead of dsubu for jump address calculation... Any more of these and I will have an overflow exception on anger. Tomorrow, or somewhere in the next week I'm doing keyboard support. Stanislaw Skowronek
Another good surprise was due in may, as Peter Fürst posted a promising
boot on a POWER Indigo2 R10000 (IP28):
Date: Sat, 15 May 2004 08:36:35 +0200 (CEST)
From: Peter Fuerst
To: linux-mips
Subject: IP28
How many Linux logins took place on a IP28-machine before ... ?
with kind regards
peter fürst
[...]
Starting up the system...
To perform system maintenance instead, press
System Maintenance Menu
1) Start System
2) Install System Software
3) Run Diagnostics
4) Recover System
5) Enter Command Monitor
Option? 5
Command Monitor. Type "exit" to return to the menu.
>> unsetenv OsLoadPartition
>> setenv OsLoadOptions "root=/dev/nfs nfsroot=192.168.1.1:/var/V/mipseb ip=192.168.1.28:::255.255.255.0::eth0:"
>> printenv
SystemPartition=bootp():
OSLoader=vmlinux
OSLoadFilename=vmlinux
AutoLoad=Yes
TimeZone=PST8PDT
console=d1
diskless=0
dbaud=9600
volume=80
sgilogo=y
autopower=y
netaddr=192.168.1.28
eaddr=08:00:69:0b:11:e2
boottune=0
ConsoleOut=serial(0)
ConsoleIn=serial(0)
cpufreq=174
OSLoadOptions=root=/dev/nfs nfsroot=192.168.1.1:/var/V/mipseb ip=192.168.1.28:::255.255.255.0::eth0:
>> boot
Setting $netaddr to 192.168.1.28 (from server Opal.Peter)
Obtaining /vmlinux from server Opal.Peter
2053216+397312+155512 entry: 0xa800000020200000
init_arch(7,9000000020ffe300,9000000020ffdf70,a800000020200000),
CP0: status 34004880, cause 0000801c, config 6c11ac03 ...
ARCH: SGI-IP28
PROMLIB: ARC firmware Version 64 Revision 0
CPU revision is: 00000925
FPU revision is: 00000900
load_mmu: R1X000 (20) ...
Primary instruction cache 32kB, physically tagged, 2-way, linesize 64 bytes.
Primary data cache 32kB 2-way, linesize 32 bytes.
Unified secondary cache 1024kB 2-way, linesize 128 bytes.
init_arch: start_kernel() ...
Linux version 2.4.22 (pf@Opal.Peter) (gcc version 2.95.4 20010319 (prerelease)) #163 Sat May 15 02:22:59 CEST 2004
Entering setup_arch()...
MC: SGI memory controller Revision 5
MC: Boardrev. 0, Chiprev. 13
MC: Probing memory configuration:
bank0: 64M @ 28000000
bank1: 64M @ 2c000000
bank2: 128M @ 20000000
Determined physical RAM map:
memory: 0000000010000000 @ 0000000020000000 (usable)
On node 0 totalpages: 196608
zone(0): 196608 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Leaving setup_arch().
Kernel command line: root=/dev/nfs nfsroot=192.168.1.1:/var/V/mipseb ip=192.168.1.28:::255.255.255.0::eth0:
Calibrating system timer... 875000 [175.0000 MHz CPU]
Using 87.500 MHz high precision timer.
Console: colour dummy device 80x25
zs0: console input
Console: ttyS0 (Zilog8530), 9600 baud
Calibrating delay loop... 174.48 BogoMIPS
Memory: 242616k/262144k available (2005k kernel code, 19528k reserved, 260k data, 100k init)
Dentry cache hash table entries: 131072 (order: 9, 2097152 bytes)
Inode cache hash table entries: 65536 (order: 8, 1048576 bytes)
Mount cache hash table entries: 256 (order: 0, 4096 bytes)
Buffer cache hash table entries: 65536 (order: 7, 524288 bytes)
Page-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Checking for 'wait' instruction... unavailable.
Checking for the multiply/shift bug... no.
Checking for the daddi bug... no.
Checking for the daddiu bug... no.
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
Journalled Block Device driver loaded
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
pty: 256 Unix98 ptys configured
DS1286 Real Time Clock Driver v1.0
SGI Zilog8530 serial driver version 1.00
tty00 at 0xbfbd9830 (irq = 45) is a Zilog8530
tty01 at 0xbfbd9838 (irq = 45) is a Zilog8530
Hardware Watchdog Timer for SGI IP22: 0.2
sgiseeq.c: David S. Miller (dm@engr.sgi.com)
eth0: SGI Seeq8003 08:00:69:0b:11:e2
SCSI subsystem driver Revision: 1.00
wd33c93-0: chip=WD33c93B/13 no_sync=0xff no_dma=0 debug_flags=0x00
setup_args=,,,,,,,,,
Version 1.25 - 09/Jul/1997, Compiled May 13 2004 at 20:54:51
wd33c93-1: chip=WD33c93B/13 no_sync=0xff no_dma=0 debug_flags=0x00
setup_args=,,,,,,,,,
Version 1.25 - 09/Jul/1997, Compiled May 13 2004 at 20:54:51
scsi0 : SGI WD93
scsi1 : SGI WD93
sending SDTR 0103013f0csync_xfer=2c
Vendor: FUJITSU Model: M2954S-512 Rev: 0142
Type: Direct-Access ANSI SCSI revision: 02
sending SDTR -REJ-
Vendor: TOSHIBA Model: CD-ROM XM-5401TA Rev: 0436
Type: CD-ROM ANSI SCSI revision: 02
Attached scsi disk sda at scsi0, channel 0, id 1, lun 0
SCSI device sda: 8498506 512-byte hdwr sectors (4351 MB)
Partition check:
sda: sda1 sda2 sda3 sda4
Attached scsi CD-ROM sr0 at scsi0, channel 0, id 3, lun 0
sr0: scsi-1 drive
Uniform CD-ROM driver Revision: 3.12
SGI HAL2 revision 0.1.0
NET4: Linux TCP/IP 1.0 for NET4.0
IP: routing cache hash table of 4096 buckets, 64Kbytes
TCP: Hash tables configured (established 131072 bind 65536)
IP-Config: Complete:
device=eth0, addr=192.168.1.28, mask=255.255.255.0, gw=255.255.255.255,
host=192.168.1.28, domain=, nis-domain=(none),
bootserver=255.255.255.255, rootserver=192.168.1.1, rootpath=
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
Looking up port of RPC 100003/2 on 192.168.1.1
Looking up port of RPC 100005/1 on 192.168.1.1
VFS: Mounted root (nfs filesystem).
Freeing unused kernel memory: 100k freed
do_execve(/sbin/init)...
INIT: version 2.74 booting
[...]
Hard Hat Linux Hard Hat release 5.1 (Copenhagen)
Kernel 2.4.22 on a mips64
indigo2 login:
[...]
He spent the next months cleaning his code, and published a first version of his
patches in august.
Date: Sat, 21 Aug 2004 02:17:40 +0200 (CEST) From: Peter Fuerst To: linux-mips Subject: IP28 Kernel patches Hello ! The Kernel patches for SGI Indigo2 R10k (IP28) can now be found at http://home.alphastar.de/fuerst/download.html with kind regards...
2004, NetBSD
Hardware support made progress in NetBSD too in 2004.
First, Steve Rumble’s code for the original (R3000) Indigo (IP12) got
merged
by Antti Kantee.
Date: 04/11/2004 16:28:32
From: Antti Kantee
To: port-sgimips
Subject: IP12 support in-tree
I've just finished pushing Steve Rumble's IP12 (4D/30, 4D/35, Indigo
R3k) code into -current (hopefully the changes will make it also into
NetBSD 2.0). Machines should be able to boot to multiuser at least using
a NFS root now (sd0 root untested, probably some funnies hiding there).
Things seem to work mostly okay, the only big issue is with nfs root
going berzerk if you try to use the default fragment sizes. Steve hinted
that using small read and write fragment sizes might do the trick,
and at least it's a working workaround for me:
dm:/m/dm/nfs/4lom / nfs rw,-w=1024,-r=1024 0 0
A kernel config file is not yet available due to changes that are going to
(hopefully!) happen before NetBSD 2.0. But for the millions of people
out there who can't just wait to run NetBSD on their IP12's, it's pretty
easy to mop GENERIC32_IP2x into a good IP12 kernel conf:
* change TEXTADDR from 0x88069000 to 0x80002000
* replace MIPS3 with MIPS1 (yes, replace, needs investigation)
* add: pic0 at mainbus0 addr 0x1fa00000
* add: gio0 at pic0
Thanks go naturally to Steve Rumble for writing most of the code and an
endless stream of answers to questions about SGI hardware, and also to
Chris Sekiya for shipping me a battery to replace the dead one on my IP12
(.. and answering questions.. and working on abstracting sgimips enough
for support to be easily dropped in.. and ..).
Enjoy! Obligatory bootlog follows.
>> hinv
Memory size: 32 Mbytes
Instruction cache size: 32 Kbytes
Data cache size: 32 Kbytes
System option: Audio processor, revision 3
CPU board: IP12 33 MHz, with FPU
System option: Audio processor, revision 3
>> boot
Setting $netaddr to 10.181.181.183 (from server dm)
Obtaining netbsd from server dm
1434800+0+170476 entry: 0x80069000
argv[0]: bootp()netbsd
[ Kernel symbol table missing! ]
Mem block 1: type 0, base 0x0000, size 0x0001
Mem block 2: type 2, base 0x0001, size 0x0fff
Mem block 3: type 2, base 0x1000, size 0x0800
Mem block 4: type 2, base 0x1800, size 0x0800
Mem block 5: type 4, base 0xfc00, size 0x0000
Loading cluster 1 (before kernel): 0x1 / 0x68
Loading cluster 1 (after kernel): 0x1f1 / 0x1000
Loading cluster 2: 0x1000 / 0x1800
Loading cluster 3: 0x1800 / 0x2000
zs channel 0 had address 0xbfb80d10
Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004
The NetBSD Foundation, Inc. All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
NetBSD 2.0C (4LOM) #367: Sun Apr 11 15:15:22 EEST 2004
pooka@brain-damage.localhost.fi:/sys/arch/sgimips/compile/obj/4LOM
total memory = 32768 KB
(0 reserved for ARCS)
avail memory = 30308 KB
mainbus0 (root): SGI-IP12 [SGI, IP12], 1 processor
cpu0 at mainbus0: MIPS R3000A CPU (0x230) Rev. 3.0 with MIPS R3010 FPC Rev. 4.0
cpu0: 32KB/4B direct-mapped Instruction cache, 64 TLB entries
cpu0: 32KB/4B direct-mapped write-through Data cache
int0 at mainbus0 addr 0x1fb801c0
pic0 at mainbus0 addr 0x1fa00000
pic0: Revision B: dblk (0x2), iblk (0x8)
pic0: cache disabled, store partial, bus drive
gio0 at pic0
Synchronous ISDN (product 0x04 revision 0x00) at gio0 slot 2 addr 0x1f000000 not configured
hpc0 at gio0 addr 0x1fb80000: SGI HPC1
zsc0 at hpc0 offset 0xd10
zstty0 at zsc0 channel 1 (console i/o)
zstty1 at zsc0 channel 0
zsc1 at hpc0 offset 0xd00
zsc1: channel 1 not configured
zsc1: channel 0 not configured
int0: cannot share interrupts yet.
sq0 at hpc0 offset 0x100: SGI Seeq 8003
sq0: Ethernet address 08:00:69:06:2b:01
wdsc0 at hpc0 offset 0x11f: WD33C93A SCSI, rev=0, target 0
scsibus0 at wdsc0: 8 targets, 8 luns per target
dpclock0 at hpc0 offset 0xe00
biomask 0b netmask 0b ttymask 1b clockmask 7f
scsibus0: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 1 lun 0: disk fixed
sd0: drive offline
sd0: sync (200.00ns offset 12), 8-bit (5.000MB/s) transfers
boot device:
root device: sq0
dump device:
file system (default generic): nfs
root on sq0
nfs_boot: trying DHCP/BOOTP
nfs_boot: DHCP next-server: 10.181.181.1
nfs_boot: my_domain=localhost.fi
nfs_boot: my_addr=10.181.181.183
nfs_boot: my_mask=255.255.255.0
nfs_boot: gateway=10.181.181.1
root on dm:/m/dm/nfs/4lom
root time: 0x4079371b
readclock: 2004/4/11/12/16/9
time.tv_sec = 1081685769, time.tv_usec = 960000
init path (default /sbin/init):
init: copying out flags `-s' 3
init: copying out path `/sbin/init' 11
Enter pathname of shell or RETURN for /bin/sh:
Terminal type? [unknown]
Terminal type is unknown.
We recommend creating a non-root account and using su(1) for root access.
# Sun Apr 11 12:17:30 UTC 2004
Starting file system checks:
Setting tty flags.
Setting sysctl variables:
Starting network.
Hostname: 4LOM.localhost.fi
Configuring network interfaces: sq0.
add net default: gateway 10.181.181.1
Adding interface aliases:
Building databases...
Starting syslogd.
Checking for core dump...
savecore: /netbsd: kvm_openfiles: /netbsd: No such file or directory
Apr 11 12:18:00 4LOM savecore: /netbsd: kvm_openfiles: /netbsd: No such file or directory
Setting date via ntp.
setclock: 2004/4/11/12/18/24
Starting rpcbind.
Apr 11 12:18:25 4LOM rpcbind: cannot create socket for udp6
Apr 11 12:18:25 4LOM rpcbind: cannot create socket for tcp6
Mounting all filesystems...
Clearing /tmp.
Checking quotas: done.
Setting securelevel: kern.securelevel: 0 -> 1
Starting virecover.
Starting local daemons:.
Updating motd.
Starting sshd.
Starting inetd.
Starting cron.
Sun Apr 11 12:19:11 UTC 2004
NetBSD/sgimips (4LOM.localhost.fi) (console)
login:
[...]
A few months later, Christopher Sekiya added
almost
complete support
for the XZ (Elan) graphics option.
Date: 07/09/2004 07:39:04 From: Christopher SEKIYA To: port-sgimips Subject: IP12/20 wscons support in-tree All, I've just committed the final bits for grtwo wsconsole support on the IP20 (and probably IP12 -- the hardware is the same). It should now be possible to use a monitor/keyboard as console. There is still more work to do -- there is no cursor, and column copy/erase is problematic; however, it seems quite useable. Enjoy, -- Chris
Unfortunately, his hardware would stop functioning shortly afterwards, and he
never got to improve this code (which contains an infinite loop he probably
was lucky enough not to trigger in his testing, which I fixed years later while
completing the work on this frame buffer for OpenBSD.)
The next week, the much more awaited driver for the onboard Ethernet interface
on the O2 was also
commited
by Izumi Tsutsui.
Subject: Re: On-board MACE MAC-110 Ethernet on O2 (on-going) From: Izumi Tsutsui To: port-sgimips Date: 07/11/2004 12:35:18 [...] > After a bunch of try and error, now the driver is mostly functional: > http://www.ceres.dti.ne.jp/~tsutsui/netbsd/if_mec-20040709.tar.gz I've committed the driver with some further enhancements (especially interrupt and cache handling). http://mail-index.netbsd.org/source-changes/2004/07/11/0001.html As noted the commit log message, this driver still needs more tests. If you find any problems, please report via send-pr(1).
[go back to the previous part][skip
to the next part]
2004, OpenBSD
As usual, Fogelström disappeared for a while and we thought again that the
SGI port would never materialize unless someone else would take the lead (and
likely port the existing NetBSD code).
But on july 30th, it looks like things would happen for real.
sooon a snapshot for you guys with SGI O2s will be ready to play with. R5K required for now though. thanks pefo. i still have to check the O2 here, but I think I have an R5k module on one of the i2 or indy. thought they might not be suit for o2, come to think of it... no, O2 and Indy modules are not compatible. damn. then i hope it's an r5k.
The promised snapshot appeared two days later.
ALLRIGHT!! Everyone with SGI O2s R5K, line up and dust them off. Snapshot ready for ftp in 2 hrs! pefo, I'll line up once I'm back home tsk, tsk... ;) if you can pay the bills and the food I'll gladly stop working and spend my time home (-: [...] OK, the SGI O2 snap is up and ready for download. Have fun!
Date: Mon, 2 Aug 2004 15:57:51 +0200 From: Per Fogelström To: private OpenBSD mailing list Subject: First SGI O2 snap now available!!! OK, for all those who waited a long time, the moment has come!!! This snap runs on O2s with R5K and i think R5K2 cpus and at least 64Mb of sdram. You will need a NIC, preferably a fxp. Integrated ethernet support is on the way. Read the README file for more info. Get it at ftp.opsycon.se/pub/OpenBSD/3.5+/sgimips when it's fresh! Have fun and report back to me! Per
Theo de Raadt seized the opportunity to
ask for
hardware donations (it never hurts to try…).
Date: Wed, 04 Aug 2004 08:43:57 +0000 From: Theo de Raadt To: openbsd-misc Subject: SGI O2 A developer is about to import an SGI O2 codebase. Anyone want that? There's a catch. We need at least 3 machines in Calgary. There's a little known rule in OpenBSD -- the "to make an official snapshot" rule -- which says roughtly [sic]: in addition to the main developers of a new architecture, at least Theo and Peter must have machines. Otherwise all of us get scared of the possibility of a new architecture languishing and creating an unsightly pit of dead code in the tree. We could use more than 3, since developers visit here once in a while, and we like to send them on their way with full suitcases. Is there anyone out there that wants to take care of this? ie. Finding some machines, and getting them to Calgary. It's a mission. I'm serious. Sometimes we ask and nothing happens. It is kind of like I am asking our user community to invest some time in getting access to some damn cheap (just check Ebay) machines and simply post them to Calgary :) People seem to ask how they can help quite often; here's an example. Why add a new architecture? SGI's mips machines are dead, aren't they? Well our expierence [sic] has been that every architecture we add has helped us find bugs in shared code that affects other architectures. (As long as it receives developer [attention] and does not rot like NetBSD's architectures do). Some very scary and major bugs have been dredged out almost automatically. In some cases, the effort is not worthwhile. In this case, I judge it to be valuable. In particular, the SGI developer will probably peek out quite a few busdma bugs, which will affect driver support on any busdma architecture. ps. m88k has been the most worthwhile example, since some insane parts of that architecture permits Miod to find a MI bug weekly.
And now that there were some concrete bits, we could start addressing one of the
most important
problems in computing: naming things.
So, should it be called "sgimips" or "sgi". Comments? will we have any other mips in foreseeable future? guess it doesn't matter for the name actually Well I doubt we'll be supporting the sgi68k or sgi88k or sgisparc machines. to be fair, I have been looking for an sgi68k for some years now, they are very very very rare. I dislike typing long names, so I think it should just be sgi. well, it was supposed to be sgi initally. sgi was very successful at buying them back to give to their sledgehammer guy. I met their California sledgehammer guy. yup I was introduced :) did he hit you with his hammer? Was told that they spent about 1 million dollars over 10 years keeping them off ebay and out of the hands of other resellers. heh. am not surprised. why bother? To kill the reseller market. SGI machines had low value on buyback. Unlike Sun, who put high value on buyback. sgi tried to be very vocal on the theme "2nd hand sgi are bought as refurbished from sgi".
After one good night of thought…
so, sgi or sgimips? I think sgi. sgi sounds good to me. sgi also sells ia64 systems isn't it? Yes, but those a standard ia64 systems. Nothing SGI about them. ah, ok Hmm, NetBSD calls their SGI port sgimips though Yes, and they call their amd64 port x86_64 [...] it seems to be more of an issue of hpcmips vs sgimips rather than sgi making any other than mips machines same as macppc vs mvmeppc [...] I like simpler. [...] sgi68k is not going to happen. right so sgi makes sense. on he other hand there is no catsarm cats have legs, not arms, dummy paws you fluffy you and tails. [...] but are sgimips and sgimips64 the same port or 2 different ones? isn't sgimips 32bit only so far? I think it is 64bit only. NetBSD/sgimips is 32bit only as far as I understand. [...] so we want 2 ports sgi and sgi64 matthieu, likely. We want to support the 32 bit machines? Or just say forget it? i'd like to. but then I have too much time on my hands as everyone knows. so maybe sgi64? or sgi for 64 bits, and sgi32 later (-: sure if. when. hell, you said you wanted to only have to type "sgi". i prefer sgi :) my vote is "sgi" == 64bits, all cpu's I own atm are supposedly 64bit capable it's "sg " anyway. So who cares?
(catsarm in the discussion above is a reference to the ARM-based
CATS board, which was used to
bring back ARM support in OpenBSD in order to have a solid fundation before
starting the
SHARP Zaurus port.)
Fogelström came with a nice summary the next day.
hey, while i have a good night sleep you people can decide wether you want to call it sgimips, sgi, sgi64, pamela, or whatever! ;)
Eventually there was a general agreement that “sgi” was the better name,
and the source code was added to the OpenBSD source tree on august 6th.
Michael Shalayeff ported the NetBSD O2 on-board Ethernet driver a few days
later, and a few other developers started working on code cleanup.
Date: Sat, 14 Aug 2004 15:40:01 +0200 From: Per Fogelström To: private OpenBSD mailing list Subject: New SGI snapshot available. New snap available in ~pefo/sgisnap-0814 This snap has mec ethernet driver and a lot of other fixes. When using mec for network be aware that there is a snag somewhere we are searching for which makes the driver hang the system. Although i have done complete make builds via nfs source using mec there is something hiding in there. fxp is reliable though. MACHINE is now sgi instead of sgimips, MACHINE_ARCH=mips64. A little confusing since code is still LP32. As a consequence of that cc and as does not agree on things and -mips1 or -mips2 has to be explicitly given as option. I will try to fix that in the next snap coming in a couple of days. Alternatively pick up the comp35 tar from ftp.opsycon.se and use binutils from there. No disk boot yet. Code is ready but not tested yet. Heading for LP64 now!!! Per
A few days later:
Date: Tue, 17 Aug 2004 07:56:16 +0200 From: Per Fogelström To: private OpenBSD mailing list Subject: SGI snap update The SGI snap in ~pefo on cvs is now updated. The toolchain should now be working (nm(1)). The only tar which is updated is the comp36.tgz. nm(1) doesn't work with binutils 2.14 and mips. It does not say 'T', 'U' etc on shared lib symbols. 'W' works though. If someone could take a look at this i would appreciate it since perl does no longer build because of that. The snap dir also contains a diff against the tree with the patches currently needed to do a make build. Many of these will be obsolete when the toolchain is fixed, among them the GOT separation and alignment. Basically all MI fixes will be gone and only some MD will be left. I will be on the mainland for a couple of day, back Thursday, so until then have fun. :) Per
Before performing the switch from 32 to 64 bit, there were a few less important
things Fogelström wanted to address, which took the rest of august: a working
standalone bootloader, and the switch from gcc 2.95 as the system compiler to
gcc 3.3, which was a requirement for reliable enough 64-bit code generation.
SGI: diskbooting now works. code is not yet committed, i have to run. a new snap will be put up later today or tomorrow. ... ok, a new SGI snap is in ~pefo/sgisnap. the kernel does not yet autodetect the boot device but the one in the next snap will. it's building right now. don't forget to 'setenv OSLoader boot' otherwise it will try to start sash. (for you who don't read the install doc ;) )
On august 31st, things were ready for the 64-bit port to start.
ahh! sgi fully migrated to gcc3 now. oh really? in mips32 or mips64? mips32. now going 64. neat.
Date: Tue, 31 Aug 2004 22:03:46 +0200 From: Per Fogelström To: private OpenBSD mailing list Subject: gcc3 based sgisnap available As usual in ~pefo at cvs. This is probably the last snap before going full 64 bit.
Two days later, a 64-bit kernel was working.
Loading ELF64 file 0x0:0xffffffff, Zero 0x339bb0:0xffffffff, 0x347bd0:0xffffffff, start at 0x801001d0 Found SGI-IP32, setting up. Initial setup done, switching console. -Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2004 OpenBSD. All rights reserved. http://www.OpenBSD.org OpenBSD 3.6 (GENERIC64) #24: Thu Sep 2 13:24:13 CEST 2004 root@moosehead.opsycon.se:/usr/src/sys/arch/sgi/compile/GENERIC64 real mem = 134217728 rsvd mem = 7020544 avail mem = 108924928 using 1638 buffers containing 6709248 bytes of memory mainbus0 (root) ... TADA!!! hip hip hurray! well this was the easy part, migrating the kernel to 64 bits. now comes userland... i'm cheating a little though... not running on full address space yet. and don't tell theo, the same code is used to build a 64 or 32 bit kernel. just feed the compiler -32 or -64 and everything is taken care of. ;) /msg deraadt did you know pefo cheats? he uses the same code to build 64 and 32 bit kernel! oops ;-)
Userland took four more days, with some setbacks.
wow! just went multiuser in 64 bit mode on sgi. userland is static since ld.so needs some fixing. but this is looking promising! [...] ssh craps out on mips64 RSA_public_decrypt failed: error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 is there something in the libs that needs to be set to 64? my guess would be libssl/crypto/arch/mips64/opensslconf.h oh! thanks! it's a great timesaver when someone knows the answer or where to look! dale was there with arm ;-) :)
This 64-bit mips work also led to unexpected discoveries.
heh! found a new lever on my chair! this one adjusts the y-position of the seat. and i've had it for almost 3 years! next week, you'll find the instructions manual!
The 64-bit adaptation work was
commited
on september 9th.
Kernel moves to 64 bit. A few more tweaks when binutils is updated.
And then we faced our first severe bug.
oh crap! panic: pool_get(mbpl): free list modified: magic=e291a1a; page 0xffffffffc332f000; item addr 0xffffffffc332f880 [...] panic: pool_get(mbpl) happens everytime i try to ssh to the O2. smells like alignment issue. i.e. you allocate @0 but access @0+4 onwards... what is mbpl mbuf pool ok. perhaps i should try using the old trusty fxp to see if it may be an mec driver problem. when i think about it's very possible. that driver have never run in 64 bit mode before since netbsd have no mips64 yet. [...] do you still get the mbpl panic? haven't had time to check that any further. i switched to a fxp though but that one crashes in the driver with a messed up mbuf chain. funny thing is that dong nfs from the O2 works fine. but as soon as i try to ssh to the box it crashes. same sypthimh taht is with the fxp. either fxp or mec works fine nfs-client. sympthom that... never saw this problem with the 32bit kernel. [...] panic: pool_get(mbpl): free list modified: magic=56617018; page 0xffffffffc332f000; item addr 0xffffffffc332f580 still not triggered early by your diff Todd )-; Oh well still a nice idea. it's from the MGET in m_prepend(). Yeah, I see miod, do you have a trace so you can see where pool_get is called ? _pool_get+0x644 (1ffffce7,ffffffff803caab0,56617018,ffffffffc332f000) sp ffffffffc6c6b770 ra ffffffff801d3f6c, sz 64 m_prepend+0xb4 (1ffffce7,ffffffff803caab0,56617018,ffffffffc332f000) sp ffffffffc6c6b7b0 ra ffffffff80271c80, sz 48 udp_output+0x130 (1ffffce7,ffffffffc3293008,0,0) sp ffffffffc6c6b7e0 ra ffffffff80272608, sz 144 udp_usrreq+0x638 (1ffffce7,ffffffffc3293008,0,0) sp ffffffffc6c6b870 ra ffffffff801d933c, sz 64 sosend+0x62c (1ffffce7,0,0,ffffffffc332f480) sp ffffffffc6c6b8b0 ra ffffffff802b7d00, sz 128 nfs_send+0x88 (1ffffce7,0,0,ffffffffc332f480) sp ffffffffc6c6b930 ra ffffffff802b9430, sz 32 nfs_request+0x9e8 (ffffffffc3304de0,ffffffffc332f000,3c6c6bdd0,ffffffffc332f480) sp ffffffffc6c6b950 ra ffffffff802cdf10, sz 288 nfs_lookup+0x400 (c3304de0,ffffffffc332f000,3c6c6bdd0,ffffffffc332f480) sp ffffffffc6c6ba70 ra ffffffff801f4308, sz 464 VOP_LOOKUP+0x60 (ffffffffc3304de0,ffffffffc6c6be10,ffffffffc6c6be38,ffffffffc332f480) sp ffffffffc6c6bc40 ra ffffffff801e8c2c, sz 48 ddb> show pool mbpool POOL mbpl: size 128, align 8, ioff 0, roflags 0x00000018 alloc 0xffffffff80434538 minitems 16, minpages 1, maxpages 8, npages 1 itemsperpage 31, nitems 20, nout 11, hardlimit 4294967295 nget 2638, nfail 0, nput 2627 npagealloc 1, npagefree 0, hiwat 1, nidle 0 currently entered from file /data/src/sys/kern/uipc_mbuf.c line 236 (sorry for flood) flood expected for a pool overflow, you know. go drown yourself. ENOMEM: pool not big enough. QUI EST GROS ? stfu, anything interesting scrolls off! ;) echo -n "heh" you get this in nfs, eh? ssh -1 machine, but the private key is on an nfs mounted share. gonna reboot and try ssh -lroot... but then nfs when logged on the console works. (and this ssh works ~ 1 time out of 3) I'm using nfs from the console too for my source tree. ah, ok. i'm strting to think it has to do with fragmentation. i get my crash in mec_start when it figures that the packet can't be send as is but must be revuilt. pefo, but you had this with fxp as well, right? it crashes with the fxp, but it looks different. may be something else although related since nfs works fine but ssh to box crashes. to bad the R5K doesn't have the watch register feature. could have nailed this in less than an hour... :( i thought you had an RM7k o2 too? no, r10k, and a r12k cpu module on its way. the r7k seems to be very rare... i have several other mips systems/boards with rm7k's though but they don't run mips64 yet. [...] R10K have the watch register. looks like i'm going to add support for it a little earlier than i planned...
(“Qui est gros?” above in all caps is an
Obelix reference.)
It took several days and many people’s brains to figure out; that was caused by
a machine-dependent constant in the network stack, which ought to have been
enlarged during the switch to 64 bit, but had been left unchanged, leading to
network stack assumptions no longer being respected.
The machine-independent nature of the bug was confirmed by testing with an
incorrect value on other platforms and experiencing similar network memory
corruption.
This was
fixed
on the 17th:
Crank MSIZE and NMBCLUSTERS, per other 64bit arches.
Further investigation of the causes of that bug also pointed out an earlier
change in the network stack had been subtly incorrect, and that change was also
reverted.
Another good side-effect of that bug hunt was that Fogelström worked on R10000
processor family
support
earlier
than initially intended.
OpenBSD/sgi (moosehead12k.opsycon.se) (tty00) login: mickey! want a new kernel? how come it sez (tty00) instead of (console) prolly something in conf.c? actually have no idea. my mind have been busy with other things. :) probably your /etc/ttys console "/usr/libexec/getty std.9600" unknown off secure tty00 "/usr/libexec/getty std.9600" unknown on secure oh, i did not notice this earlier. go ahead and fix it if it's wrong. no, actually it's ok until we get video console and that will be??? ;) hopefully soon you have the stuff from glaurung's? i have looked at it, yes and then my hair turned white. miod, but you didn't lose your hair! it could have fallen off! ;) now we know why Miod wears a hat i'm running the 12K with dirty speculative still on in kernel mode, eg like the 10K will do. building a kernel right now and so far it seems to work OK. actually, i should get a haircut soon. just powered up the Origin 200. pretty beefy machine. 225Mhz, 512Meg. only 225 MHz? that's a scam! for a R10K that is pretty good. but my fastest r4.4k runs at 250MHz! (ok, I'm sounding like an old record again here) you will be outrun anyway ;) building a kernel with the R12K was a little more than 3 times faster. i had hoped for about 4, but anyway. wait till you have smp code working! (-: oh yeah! and 1 128 cpu Origin 2000 cluster! Only $10K on ebay! ;)
(glaurung in the discussion above being Vivien Chappelier.)
(Also note the name of Fogelström’s system – Moosehead was the SGI code
name for the O2, and 12k obviously is the processor type, a
MIPS R12000.)
The mips64 codebase would enter another turbulence zone, and this time I was the
one to blame. The machine-dependent part of the virtual memory subsystem, known
as the pmap module, had still some parts coming from the OpenBSD/arc code years
ago, and was behind many changes, in particular the data structures handling
the modified and referenced state of the virtual memory pages (which had to
be maintained by software on MIPS) could be improved. While working on these
changes, I tried to kill too many birds with a stone (my first mistake) but
did not test well enough (my second mistake) and introduced several bugs
which caused, among other things, random segmentation faults in userland
binaries.
I should have reverted my changes and done them again in smaller steps, but I
was so sure this would be fixed by minor changes (a “one-liner” or two) that I
did not want to do it (my third mistake); it took pressure from several
developers and a heated discussion with more curse words than should have been
needed for me to revert the troublesome parts, and we had lost three days.
But this allowed a much more stable snapshot to be built and released on the
next day.
Date: Fri, 24 Sep 2004 13:30:18 +0200 From: Per Fogelström To: private OpenBSD mailing list Subject: New sgi snap available. OK, after much bug digging a new snap is put up in ~pefo@cvs. The kernel now seems stable wrt random core dumps. Problems were found and fixed in pmap code and in ld.so. This snap still lacks sendmail and friends. gcc still barfs on it. binutils is a moving target and seems to have bugs which manifest themselfs as failed linking of certain programs. it's being looked at. however it means that a few programs fail to link or cores. Most of these can be linked static but the major mess is gcc. don't try to rebuild it unless you really need (wanna fix bugs?). in that case contact me and i will explain how to build it. however if something cores, try to build it -static. groff for example must be linked static. There are two extra files in the snap dir. One is the emulparam that is going into ld. You only beed this if you plan to rebuild binutils and especially ld. The other file contains the diffs i have wrt head in my build tree. mostly binutils but also a "gross" hack to ahc.c to achive full disk speed. A better fix for this is coming. Note that this diff is not MI safe so be careful. known problems beside those in the toolchain is that the mec ethernet chip sometimes get stuck with its interrupt asserted. a power cycle or a reboot from the maintenance console fixes it. ahc craps out now and then, it seems. i'm not sure if this may be related to the R10K speculative dirty problem. it would be nice if people could test on both r10k's _and_ other machines to see if the problem occurs over the entire line.
(The “R10k speculative dirty problem” is a reference to the speculative
execution behaviour on this processor. Refer to the
technical note I wrote earlier
about it for details. In both NetBSD and OpenBSD, the cache invalidation
discipline in the drivers, done as part of the
bus_dma layer, turned out
to be good enough to not suffer from speculative execution side effects.
Linux, on the other hand, never was that lucky, and support for R10000
O2 has never been considered stable, to the point that you
need to go out of your way in order to be able to build a kernel supporting
that particular hardware configuration.)
Bugfixes kept coming, but we had to disable the stack-protector code
on a few binaries (the libsmutil part of sendmail) as it would cause internal
compiler errors.
Eventually Theo de Raadt was able to take over the snapshot builds
in early november.
latest sgi snap is by me :) theo, you mean the latest sgi snap is unreliable (-: ? you said you built it yourself!
Later that month, the O2 I had been using (courtesy of Wim Vandeputte,
who had bought that machine and lent it to me earlier that year) failed.
damn! looks like the o2 here died amber light, no startup sound... no cereal? mine did that, but it wasn't dead - there's a jumper you should try out it's pretty close to the cpu when you take the board out, towards the edge, a single jumper used to reset everything to defaults no, peter, that was because you did a setenv of a variable wrong miod, there is a table that says what colours at boot mean what i know for amber, i have managed to let it sit off for an hour, and it worked again yeah, i'm getting old as i forgot this so quickly kind of scary [...] well, my o2 arrived completely dead. After cleaning the motherboard <-> chassis connector even the disk works now. unfortunately it is clean. i had cleaned the box when it was DOA, but here it just doesn't want to restart after being shut down one more time... it's that 10mbit crap you are hooking it up to no, it says "cpu board failure" i'll let it rest the night. wim? yo do you remember who is the guy who lent the O2 which is at my place at the moment? yes, that would be me no, you told me you got it as a lent [sic] from someone else. no, I bought it last year oh. want to attend the funeral? you broke it? it died on me, apparently. amber led? yes. not blinking. Open it up, try to fix it. Otherwise, I'll take it back and send it to my O2 doctor in .nl according to http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=hdwr&db=bks&fname=/SGI_EndUser/books/O2_OG/sgi_html/ch04.html&srch=o@%20troubleshooting it's the cpu board. wim, been there, done that. and it's not the first time this happens, but today nothing seems to bring it back to life. err, i said "cpu board failure" it's "system board failure" actually. RIP
Fortunately, I had connections at the
local university, and I was able to
get another O2 motherboard loaned to me one week later,
thanks to François Delobel and David Delon.
fall 2004 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP12 | Indigo (R3000) | complete distribution no graphics |
||
| IP20 | Indigo (R4000) | complete distribution no graphics |
||
| IP22 | Indigo2 | complete distribution XL (newport) graphics only |
complete distribution no graphics |
|
| IP24 | Indy | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only, no X server |
|
| IP27 | Origin 200, Origin 2000 | complete distribution | ||
| IP28 | POWER Indigo2 R10000 | not-yet-integrated kernel patches otherwise same as IP22 |
||
| IP30 | Octane | not-yet-integrated kernel patches X server on Impact only |
||
| IP32 | O2 | not-yet-integrated kernel patches no R10000 support |
complete distribution no graphics |
complete distribution no graphics not public yet |
2005, OpenBSD
I spent some time trying to get the O2 frame buffer to work, with no success.
At some point I vented my frustration.
Date: Fri, 18 Feb 2005 21:40:02 +0000 From: Miod Vallat To: private OpenBSD mailinglist Subject: O2 video Ok, this is a nightmare. It turns out the Linux driver has been written after noticing the O2 ``GBE'' hardware is close to the SGI ``DBE'' found in the expensive x86-based ``Visual Workstations'' they produced in '99 or so. These chips are supposed to be smart because they have no frame buffer memory, but instead do DMA blits from the main memory, laid out in ``tiles'', in order to allow the memory to be discontiguous in practice. Just like on Zaurus (-: (except the Z uses a single contiguous area) So the Linux guy tinkered a bit, got something close to working, tinkered more, and tada, it was deemed working. It is obvious, though, that he never looked at the contents of all the ``DBE'' registers first - because the GBE layout is slightly different. In particular, a few things in the Linux driver are clearly wrong, but apparently they are lucky enough to not suffer from putting apples in the pumpkins registers. Too bad I have not been as lucky, so either with ``more correct'' code or with the exact same sequence of operations as the Linux code, I lose - either an unstable image or a nice black screen. Or an interleaved madness which I can not make any use of )-: (not to mention spurious interrupts, but this part is solved now). I am trying to find more documentation about the GBE, and will probably disassemble some IRIX code to help... Maybe there are different revisions of this piece of hardware as well... Anyway, knowing that breakthroughs usually happen *after* I send mails about non-working code or problems I am stuck with, I wrote this mail only to relax my mind and shuffle my ideas, in the hope of getting the damn thing to work. If you have read till this sentence, you may discard this mail and resume your regular slack^Wwork. Thanks for reading! Miod PS: Several versions of the non-working code are available upon request if you want to play this game, too!
One month later, I had to return the O2 motherboard and was unable to do
further tinkering. An Octane was lent to the OpenBSD project, and ended up
at my place in may, but I had no time to tinker with it either.
I relocated across the country in autumn, and thanks to the help of Matthieu
Herrb, I was lent another O2 and could resume bug hunting on that platform.
Near the end of the year, I stumbled upon a funny bug introduced during the
switch from 32 to 64 bits: in userland, there were implicit memory aliases every
2 GB. In other words, if you had a variable at address A, accessing memory at
address (2GB + A) would not only not cause a segmentation fault, but would
return the value of the variable. I wrote a simple program demonstrating this
fact, and shared it with the appropriate kernel fix.
Date: Fri, 16 Dec 2005 07:37:59 +0000 From: Miod Vallat To: Per Fogelstrom, private OpenBSD mailinglist Subject: [mips64] userland space aliases The current mips64 codebase has an oddity introduced when switching from 32 to 64 bit: in userland, all addresses between 0 and 7fff.ffff.ffff.ffff are aliased every 2 GB. Let's consider the following test program: #include#include #include #include jmp_buf cheat; void segv(int signo) 💬 void test(char *address) 💬 int main() 💬 On mips64, here is what you will get: $ ./obj/amazing accessing 0x7ffe0720... 0 accessing 0xbffe0720... segfaulted! accessing 0xfffe0720... 0 accessing 0x13ffe0720... segfaulted! accessing 0x17ffe0720... 0 accessing 0x287ffe0720... 0 $ Yet the userland address space (so far) is supposed to be restricted to 2GB... The reason behind this is that the XTLB refill handler has been cloned from the 32bit TLB refill handler, but needs more bounds checking. In the 32bit world, the current pmap scheme makes sure that all virtual addresses (0 to ffff.ffff) are managed in the pmap's pm_segtab. But in the 64bit world, there is a hole between the 2GB userland limit (at 0000.0000.8000.0000) and the kernel space (at 8000.0000.0000.0000), which is not checked by the TLB handler. Since the code does a logical and operation to narrow the pm_segtab access, this means the upper bits of the logical address are ignored. In practice, this means any access to a particular address would end up using the mapping for this address modulo 2GB. The diff below is a suggested fix to this - we simply add this bounds check in the XTLB refill handler. Note that the TLB refill handler does not need to be modified, as it will only get invoked for faults in the 32-bit address space, where we are always within our bounds. The test program will thus behave as expected: $ ./obj/amazing accessing 0x7fff4af0... 0 accessing 0xbfff4af0... segfaulted! accessing 0xffff4af0... segfaulted! accessing 0x13fff4af0... segfaulted! accessing 0x17fff4af0... segfaulted! accessing 0x287fff4af0... segfaulted! $ Comments? Miod [...]
The fix was
commited
shortly afterwards.
2006, NetBSD
There was not much visible sgi-related activity in NetBSD in 2006.
During the second half of the year, Steve Rumble added support for some Fast
Ethernet GIO expansion cards for Indigo, Indy and Indigo2, as well as
for the LG (Indigo entry-level) frame buffer.
2006, OpenBSD
2006 was a quiet year for OpenBSD on the sgi front. I had noticed some odd
things in the kernel and stumbled upon worse problems every time I tried to
clean or fix them.
At the end of october, given the stability issues on the rise on the
O2, Theo de
Raadt considered pulling the plug.
[go back to the previous part][skip
to the next part]
2007, NetBSD
There was not much sgi-related activity in NetBSD in 2007.
The only important accomplishment was the addition of a
driver
for the O2 frame buffer
in april, thanks to Jared McNeill‘s hard work.
Date: 04/12/2007 09:25:47 From: Jared D.McNeill To: port-sgimips Subject: Testers wanted: SGI O2 framebuffer console and keyboard controller in -current Heyas folks -- I've added wscons support to the GENERIC32_IP3x kernel config in - current. Feel free to bang away at it and let me know how it works out for you :) The framebuffer driver will configure your display using whatever your firmware setup by default. It will also steal 4MB of physical memory from you -- I'm looking into ways of reducing this. As for XFree86, if you manage to get the wsfb driver built, the display driver will support it. It does seem that bus_dmamem_mmap ignores any hints passed to it, and the result in X is quite interesting :) Cheers, Jared
That work was completed by Michael Lorenz to provide a
true
colour X server
in july.
Date: 07/25/2007 23:56:16 From: Michael Lorenz To: port-sgimips Subject: X on O2 Hello, I just committed a few changes to crmfb and wsfb - now you can run XFree86 with the wsfb driver in 24bit colour. Still no acceleration but since the framebuffer is in main memory it's not horribly slow either, in fact I've seen worse performance with hardware acceleration on some machines... There's one problem though - gtk2 apps won't work because the underlying graphics library - Cairo - apparently doesn't support the pixel format used by the O2's graphics hardware. And apparently nobody expects to encounter something like that so Cairo passes a NULL pointer to Xrender which doesn't check it either so there's your SEGFAULT. KDE, gtk1, Motif etc. work fine though. have fun Michael
2007, OpenBSD
Noone could realize this at the time, but the events which would unfold into
OpenBSD/sgi getting support for the Fuel started on march 12th.
friend of mine bought an SGI Fuel (r14000-based), asked me if there was a chance of BSD running on it...
The next month, Saâd Kadhi gave me his O2.
He told me he had never been able to
get OpenBSD to run on it, and I first suspected an incompatibility in its
PROM, which was a much more recent version than the O2 we were used to run
OpenBSD on. But once I could tinker on his machine, it turned out that he had
set up his partitions incorrectly and ended up overwriting the boot loader code.
That machine was powered by a 300MHz RM5200 processor with 1MB of L2 cache, and
was an appreciated performance improvement over the 180MHz R5000SC with only
512KB of L2 cache I had been using until then.
That machine also came with 640MB of memory, but OpenBSD would only
report 256MB. This was a known problem, as Mark Kettenis‘ O2 had the same
behaviour albeit having 448MB, but nobody had done any serious work to address
this limitation.
With such a machine at my fingertips, I had no excuse not to give this task a
try. But first, I needed to work on the system overall stability.
Thankfully, I identified and corrected an
horrible bug
in the way page table memory was managed, which could be blamed
for a lot of the instability problems seen recently.
oh sgi is way more stable now? yes well we will see when i get home if it still crashes for me ;) both jsg's r5k and the new rm5200 here are happy
A few days later, while working on supporting more than 256MB of memory, I
realized the kernel code had yet another horrible bug, where the value of an
important processor register would not keep the value carefully programmed upon
kernel startup.
Date: Mon, 23 Apr 2007 20:48:39 +0000 From: Miod Vallat To: Theo de Raadt, Mark Kettenis, Jonathan Gray, Per Fogelström Subject: [mips] to psr or not to psr The current code does not maintain a consistent value for the ``status register'' accross the life of the kernel. Because of a couple issues, we end up having the SR being zero (or, well, with all the nice bits clear) most of the time. This goes in the way of two things: - r12k systems are supposed to run with the DSD bit set all the time. - my work towards a real 64bit (well, 40) address space requires SR_KX to be set for the large addresses to be processed correctly (guess how I found out this problem). The following diff tries to keep SR at a better value all the time: - when saving FP context, ``or'' values into it, rather than ``set'' them. - when forking a new process, make sure it will restore the SR from its parent in proc_trampoline(). - initialize proc0 SR to what it was in the kernel after early initialization. This diff should apply cleanly whether you run with some of my pmap diffs or not. This works for me (I am running a kernel with all direct mappings but proc0 stack in XKPHYS at the moment, which means I can enable more than 512MB of memory - that's tomorrow's work). Miod [...]
With a few more days of work, and improving on a memory detection diff written
by Mark Kettenis querying the memory controller instead of asking ARCBios for
that information, I could apparently use the whole memory of the new
O2.
kettenis, your december 2005 diff was not in vain: OpenBSD 4.1-current (GENERIC) #58: Wed Apr 25 20:19:57 GMT 2007 miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/GENERIC real mem = 671088640 rsvd mem = 7020544 avail mem = 587165696 using 5734 buffers containing 33554432 bytes of memory mainbus0 (root) cpu0 at mainbus0: PMC-Sierra RM52X0 CPU rev 10.0 300 MHz with RM52X0 FPC rev 10.0 it took you almost one and half year to test? three digit clock speed. he overcompensated. :) is he cross compiling from the 68k box again? no, it took me one and a half year to write the infrastructure for > 256MB to work. [...] wow, you have more memory than I in your sgi actually i thought it had 768MB, to be honest. i'll have to recheck and matthieu's has 448 or 512 borrowing memory from his, i could reach the max 1GB heresy!
I shared my work two days later.
Date: Fri, 27 Apr 2007 18:36:05 +0000 From: Miod Vallat To: private OpenBSD mailinglist Subject: sgi > 256MB The following diff lets your O2 use the whole memory you blessed it with. This is a small starting step, where the virtual memory layout is not modified yet: kernel is limited to 1GB KVM, userland to 2GB address space. This will improve over time. Also, to keep the changes minimal, I did not replace all instances of KSEG0 or KSEG1 operations with XKPHYS operations. These will be replaced on a ``i need to make other changes to this file'' basis. Note that there is nothing wrong with cache flushes using KSEG0 addresses, since the cache aliasing mask is much smaller than the 256MB KSEG0 and KSEG1 let you access. Special care is necessary in mem.c to allow /dev/kmem access to physical memory with either KSEG[01] or XKPHYS. And of course, it is necessary to set a few more flags in the status register (which is why I had to fix its propagation accross userland a few days ago). The code checking the crime registers on O2 to find the memory above 256MB has been written by kettenis@ 16 months ago, and I only made minor modifications to it. Tested on a 640MB RM5200 and a 320MB R5000 machines. Miod PS: Do not attempt to recompile libkvm until you have installed new includes. [...]
This work ended up in the OpenBSD source tree on may 3rd.
In the meantime, I had also spent some time trying to understand why Matthieu
Herrb‘s O2 would randomly crash or hang, while another
O2 with a similar
configuration would run reliably. Eventually, I realized that the particular
R5000 processor on his machine was requiring a
processor
errata workaround,
despite being supposedly safe from that particular errata.
Edge cases can trigger a TLB miss exception instead of an invalid TLB exception on early R5000 revisions. Despite this bug being supposedly fixed in R5000 revision 2 onwards, it nevertheless occurs quite frequently on matthieu's revision 2.1 R5000. Servicing the TLB miss exception would cause a duplicate TLB to be inserted, which causes the processor operation to become unpredictable (but lethal to the kernel, ten times out of nine). More details about the problem can be found in: http://www.linux-mips.org/archives/linux-mips/2000-02/msg00040.html We work around the issue by checking for an existing TLB entry, and handling this as an invalid TLB exception (as it was intended to be), in this case. Unfortunately this causes a measurable 1% slowdown on ``safe'' processors, so we'll work on providing different tlb handler flavours in the near future to recover from this.
The important part of the linux-mips message I was referring to is:
Sometimes you get a utlbmiss exception when there is already matching
TLB entry. If you then blindly drop in the TLB entry, you get a duplicate,
which leads to Bad Things (tm). The workaround is to probe for a duplicate,
and skip the tlbwr if an entry already exists. It should be enabled on any
real R5000.
This is from the R5000 Errata list of 30 October 1997:
----------------------------------------------------------------------
3. An erroneous JTLB miss exception will be taken under
these conditions.
a) An instruction which does not cause an exception or
stall is 8 bytes away from the end of a page.
b) A load or store instruction is the last instruction of that page.
c) The load/store target address has a matching but invalid
JTLB entry
d) The next sequential page is not mapped in the JTLB
In this situation, when the load/store instruction is executed,
a JTLB invalid exception should be taken, instead a JTLB miss
exception is incorrectly taken. If the exception handler
does a random TLB write to resolve the exception, this will in
general insert a duplicate TLB entry for each erroneous exception.
If the first instruction is a jump or branch, this will cause
an infinite loop of JTLB miss exceptions to occur upon the return
from the exception handler. Otherwise, there will be only one
erroneous exception, followed by a correct exception, leaving
one duplicate entry in the TLB.
A software fix is for the JTLB miss handler to detect this situation,
by probing for a matching TLB entry (treating a hit as being this case),
ignore the JTLB miss and treat the exception as an JTLB invalid exception.
Errata 3 is fixed in Rev 2.0.
----------------------------------------------------------------------
It is not clear to me that Errata 3 is fixed in all cases in Rev 2.*,
so IRIX has the workaround enabled for all R5000 revisions.
(emphasis mine.)
At this point, the only apparently remaining problem was that, on warm boot,
the kernel would sometimes get stuck at the end of device detection, unable
to run userland, and noone had found the reason why yet; it looked like an
interrupt not being handled correctly, or not at all.
Yet, shortly afterwards, I started to experience strange, non-deterministic,
failures. Although, in retrospect, the “large memory” change appears as the
obvious suspect, these failures did not appear immediately afterwards, so I
suspected other kernel changes first, and lost time until I tried to limit
memory to 256MB.
[all caps expletive deleted] sgi works again if you disable memory above 256MB. something got broken? i don't know. there must be something left which uses the compat direct mapped segments and which i did not see so memory above 512MB logical loses might explain why the sgi with 440 MB was happier than the sgi with 640MB. hmm, mine has <512MB too IIRC i'm compiling a kernel from scratch to confirm it is ok and i'll commit the workaround
And a few hours later:
oh that's just plain gross. on sgi, dma from pci devices to memory above 256MB gets endianness-translated to get buggy drivers work? no, because the pci bridge has two windows to map physical memory, one untranslated, one translated. and the currently dumb bus_dma implementation does not know this, and translates 1:1, but it turns out physical addresses for the memory above 256MB end up in the translated window i have a simple fix for this, i just need to polish it. endianness-translated?????? well every 32 bit chunk gets swapped, apparently. scary, eh? [...] ok, who wants to read an evil sgi bus_dma diff?
A better explanation of the problem can be found in the mail I sent:
Date: Sun, 17 Jun 2007 19:50:25 +0000 From: Miod Vallat To: Jonathan Gray, Mark Kettenis, Dale Rahn, Jason Wright, Theo de Raadt, Per Fogelstrom Subject: (sgi) bus_dma diff, comments sought The following diff updates bus_dma on sgi, so that it can work with the memory over 256MB - the current scheme is not good enough. Even if you don't do mips stuff, please read on while I explain what the diff does and why it does it, because I am interested in your opinion, as a bus_dma expert, on this diff. On O2, the situation is as follows: - cpu accesses the first 256MB of memory at address 0 (this is an ArcBIOS requirement), and the rest of the memory at 1GB + 256MB onwards. Thus, we have two contiguous zones. - non-pci onboard devices access physical memory at 0x40000000 (1GB) onwards, as a contiguous area. - pci devices access physical memory at 0 onwards, as a contiguous area, unless you want endianness translation of your data, in which case you need to access at 0x40000000 (1GB) onwards. The existing bus_dma code only copes with an offset between cpu and pci view of the memory. It was using zero. As long as you do not have more than 256MB in your machine, everything works fine (we don't have any non-pci device doing dma). When I started to play with memory above 256MB, I have been pretty lucky since no I/O ever crossed that boundary. But with the change in buffers allocation, this is no longer the case, so large compilations would hit the > 256MB zone, and read reversed data. Kaboom. [...]
After some tweaks, that fix was commited on june 21st, and from then on
support for more than 256MB of physical memory was really reliable…
…at least on some machines. For some reason, Matthieu Herrb’s O2 would still
not even boot multiuser without hitting random process failures. Investigating
and tinkering, I identified issues in the interrupt handling code, but could not
yet devise a reliable way to address them, and got lost in my changes.
sgi doing its third make install in a row, i think I narrowed the issue. always the optimist? :) sgi fails on matthieu's secondary r5k, but then both Matthieu and I are suspecting the hardware - before I introduced the R5k tlb errata fix it would not even boot multiuser, if at all. well, i am not sure i have a decent grasp of the issue, but the changes I have made are consistent with the failure symptoms. it's yet another case of, when you see the workaround, it becomes obvious when you look at the symptoms. except i only had the symptoms. $ ll sgi-intr-wip* -rw-r--r-- 1 miod dmg 45923 Jun 26 18:44 sgi-intr-wip.vari -rw-r--r-- 1 miod dmg 46286 Jun 27 20:07 sgi-intr-wip2.vari -rw-r--r-- 1 miod dmg 48189 Jun 30 16:37 sgi-intr-wip3.vari -rw-r--r-- 1 miod dmg 48118 Jul 1 14:01 sgi-intr-wip4.vari -rw-r--r-- 1 miod dmg 49533 Jul 2 05:09 sgi-intr-wip5.vari -rw-r--r-- 1 miod dmg 63038 Jul 6 15:06 sgi-intr-wip6.vari -rw-r--r-- 1 miod dmg 62231 Jul 8 19:37 sgi-intr-wip7.vari -rw-r--r-- 1 miod dmg 60186 Aug 8 16:58 sgi-intr-wip8.vari one of these is the real fix problem is, 5 or 6 of this diffs cause clock issues and i do not remember which ones are safe so it's post 4.2 4.2 will only have a workaround. an ugly one. i think #5 or #6 is the best. #7 and #8 are totally impredictable but safe - except for time handling. #8 can cause time to go backwards and it's not really something you want (-: how about the first half of 3 and the second half of 6? :)
I kept tinkering with no success, and voiced my frustration about this.
Date: Thu, 16 Aug 2007 16:13:08 +0000 From: Miod Vallat To: a few OpenBSD developers Subject: Re: sgi stability diffs. good ones. really. Sorry guys, don't lose time on this, it does not work. The same kernel can make a build and a release and whatnot and be rock solid for days, then you reboot your machine with the same kernel and it won't last 10 minutes. That's so depressing I am considering changing the hostname of my O2 to ``marvin''. Miod
(The “marvin” name is, of course, a reference to
Marvin, the
Paranoid Android
appearing in
Douglas Adams’
Hitchiker’s
Guide to the Galaxy.)
Due to this sorry state of affairs, OpenBSD/sgi was not part of the OpenBSD 4.2
release.
Once the release cycle was over, we took the opportunity to invite
Joel Sing, who had been contributing good fixes to the O2 support in the
last few months, to join the project.
any new developers to setup? ajacoutot should have mailed you about Andry Breuil (spelling ?) and I think miod also backs this up. I back Landry Breuil up he's setup. next? i need to talk to Joel Sing next week he has sent sgi diffs in the last few months and is helping testing stuff
By sheer luck, at the same time, Artur Grabowski was reworking the
process switching context code in order to help future SMP improvements, and his
change had the side-effect of putting sgi back on track.
[=Topic=] miod changed the topic to "plz test switchto.9 okthxbye"switchto.9 is in snaps for the fast 5 arch's so far. excellent. it looks like it unfscks sgi as well. WHAT! what i wrote.
Joel Sing accepted the invitation and, within days, contributed a
fix
for the interrupt storm at warm boot.
Disable timer/compare interrupts on the macebus. This prevents interrupt storms from occurring on IRQ 6. ok miod@
By the end of the year, he had also written a working frame buffer driver for
the O2, allowing these systems to be used with glass console on december 14th,
with a few more fixes on the 31st.
fall 2007 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP12 | Indigo (R3000) | complete distribution | ||
| IP20 | Indigo (R4000) | complete distribution | ||
| IP22 | Indigo2 | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only, no X server |
|
| IP24 | Indy | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only, no X server |
|
| IP27 | Origin 200, Origin 2000 | complete distribution | ||
| IP28 | POWER Indigo2 R10000 | not-yet-integrated kernel patches otherwise same as IP22 |
||
| IP30 | Octane | not-yet-integrated kernel patches X server on Impact only |
||
| IP32 | O2 | complete distribution no R10000 support |
complete distribution | complete distribution (but no stable release at the moment) |
2008, OpenBSD
An SCSI controller driver tweak to let disks in the O2 run faster
than in synchronous, 10MHz mode (default wide SCSI speed, although the
controller was capable of faster transfers) was ported from NetBSD in january.
Interestingly, the issue had been noticed in the very early days of the
O2
port but then completely forgotten – in fact I only rediscovered the following
conversation, which took place in september 2004, while working on this
narrative.
how fast can an aic7880 go in sync mode? hmm 40MB it seems. 20Mhz. Negotiates at only 10Mhz. the O2 bios is not setting up the scsi at full speed. tweaking setup in ahc_pci.c allows it to run in sync 20Mhz and disk speed is much better. so using bios setting is not a good replacement for an eeprom here. 267934720 bytes transferred in 20.183 secs (13274662 bytes/sec) and it is not a very fast drive. 268435456 bytes transferred in 7.424 secs (36157628 bytes/sec) soo.. how do we feed ahc_pci.c what we want it to set up with? what is it ? hmm, wonder if this is why ahc is not too fast on macppc. (slightly older controller/drive -> ~11MB/s 'ahc0: Host Adapter Bios disabled. Using default SCSI device parameters' default pars seems to allow sync negotiations at least. but the 7880 falls back to 10Mhz in my case since it does not see "precision resistor termination".
With the O2 situation having been much improved, and a few occasional bugfixes
as bugs were found, it was time to finally work on the Octane.
I initially thought that the Octane would be quite similar to the O2, although
with more processing capabilities and a 64-bit firmware. On that specific topic,
its 64-bit flavour of ARCBios, now called ARCS, was indeed apparently providing
the known ARCBios interface, with 64-bit values. However, the fact that ARCBios
on the O2 was restricting the hardware information it exposes to only 256MB of
memory was a serious hint that pure ARCS information might not be enough to
correctly recognize the onboard hardware.
(The ARCBios interface consists of a special area in memory at a fixed
address, with a signature marker and a set of function pointers the operating
system can invoke to get information about the system characteristics, and also
perform some basic functions, such as console output, disk I/O,
getting the current time, or restart or powerdown the machine.)
Another interesting thing, specific to the Octane, is that the physical memory
is not configured from physical address zero onwards, but from address
0x2000.0000 onwards, i.e. 512MB.
And due to the way the MIPS Memory Management
Unit works, it is not possible to run a 32-bit kernel without having some parts
of the physical memory addressable within the first 512MB;
therefore there was no choice but load a 64-bit kernel.
Since OpenBSD/sgi used 64-bit binaries, this
would not be a problem, except for the bootloader code eventually, as our
bootblocks were specifically built as a 32-bit binary.
I don’t remember who, of Joel Sing and Jasper Lievisse Adriaanse,
obtained early Octane and Origin 200 code from Per Fogelström (which, similarly
to his initial OpenBSD/sgi port of 1997, had been lingering and kept private
for years.)
But this got us (well, at least Joel and I) in the mood to spend some time
working towards Octane support.
Date: Fri, 15 Feb 2008 18:44:25 +0000
From: Miod Vallat
To: Joel Sing
Cc: Jasper Lievisse Adriaanse
Subject: Re: o200 + octane code
[...]
I have started to look at pefo's bits, linux bits, and a few other
things.
First, pefo's tarball does not contain any M (all modified files have
been commited since), so what matters are the six new files:
include/mnode.h
localbus/xbowmux.[ch]
pci/xiopci💬
sgi/sginode.c
xiopcibridge is not important at the moment, because it's a
search'n'replace of macepcibridge with the O2 specific parts removed
(and actually, due to a typo, it won't attach).
These files are a very humble start, they give us knowledge of some of
the xbow structures in memory, and how to parse them.
So I gathered some information... and I am sharing the important bits.
You might want to have a look at that document:
http://techpubs.sgi.com/library/manuals/3000/007-3410-001/pdf/007-3410-001.pdf
to get more familiar with the xbow.
Skipping not-so-important details, you can consider that the system is
built of boards connected to the xbow. A board can be either a cpu board
with up to two processors, or a xio i/o board.
Let's concentrate on a single-mobo origin or octane for now. The system
will sport a mobo, and at least one xio board: the pci bridge. To the
pci bridge is connected at least an IOC3 chip, which drives the serial
ports, the keyboard and mouse ports, the parallel port, and the onboard
ethernet.
Fortunately, the IOC3 has all of these devices memory-mapped, which
allows us to access the serial ports before the xio stuff initializes.
Well, that's good, because this is all we've got now, nothing else.
Pefo's code will be able to attach two serial ports and nothing else.
What needs to be done:
1. bring the new files in and get a kernel to be loaded by the machine
2. add quick ip30 bits to match the ip27 bits so that I can work on
Octane.
3. xbow enumeration code (find xbow ``widgets'', decide what to do with
them).
4. early pci work
5. early ioc3 work, so that com attaches to ioc, not to xbow
6. xbow interrupt code. there are 2x64 interrupt bits per cpu (two
levels of hardware interrupts with a 64 bit interrupt mask), so this
will be adapted from mace.
7. fix bugs until isp@pci attaches.
8. built-in ethernet.
9. snapshot.
10. smp (-:
I'll try to work on the first five points this weekend (while I'm in the
mood).
My initial idea of the device tree for these machines would be:
xbow0 at mainbus0
xio* at xbow0 # xio connections
xiopcibr0 at xio? # pci bridge
pci* at xiopcibr?
isp* at pci? # on-boart scsi
ioc* at pci? # IOC3 superio chip
com* at ioc? # serial ports
radone* at pci? # audio
Oh, and I forgot:
5b. get and set date and time.
Later,
Miod
This was followed, the next day, with:
Date: Sat, 16 Feb 2008 21:54:30 +0000
From: Miod Vallat
To: Jasper Lievisse Adriaanse, Joel Sing
Subject: octane diff of the day
Not much success today, unfortunately.
There is indeed a kernel load address issue - I expected I could cheat
and find some address outside of already used memory areas on
ip27/ip30/ip32, but it appears there isn't any, so we'll be stuck with
different kernel configurations, unfortunately.
This diff is made of a few files from pefo coerced into compiling, and
some preliminary address work, to let uncached mappings be correct on
xbow-based machines.
The current hurdle is the inability to tell a machine is an IP30
(Octane). However, since IP27 will return a NULL pointer to the system
description call, while an IP30 won't, we can tell them apart. But the
code which finds an IP27 fails on Octane, so I am currently stuck in
arcbios.c.
I have been picking at Linux a bit (not enough), and there are indeed
significant differences between IP27 and IP30 (different clock chip,
different interrupt handling, and probably more).
I'd like to know if a kernel with the following diff boots (only to find
a serial console and nothing else) on an Origin 200 machine. Jasper, can
you give this a try?
Miod
PS: for reference, here is the hinv -t output on the Octane:
system ARC SGI-IP30 key 0
processor CPU MIPS-R10000 key 0
processor FPU MIPS-R10010FPC key 0
cache primary icache 32 Kbytes (block 2 lines, line 64 bytes)
cache primary dcache 32 Kbytes (block 2 lines, line 32 bytes)
cache secondary cache 1024 Kbytes (block 2 lines, line 128 bytes)
processor CPU MIPS-R10000 key 1
processor FPU MIPS-R10010FPC key 1
cache primary icache 32 Kbytes (block 2 lines, line 64 bytes)
cache primary dcache 32 Kbytes (block 2 lines, line 32 bytes)
cache secondary cache 1024 Kbytes (block 2 lines, line 128 bytes)
memory main 128 Mbytes
adapter XTalk heart key 0
adapter PCI key 15
adapter multi function ioc3 key 0
controller network ef0 key 0
peripheral network ef0 ethernet (100/10 base-T) key 0
controller serial IP30 tty key 0
peripheral line key 0
controller serial IP30 tty key 1
peripheral line key 0
controller pointer pcms key 0
peripheral pointer key 0
adapter SCSI QLISP1040 key 0
controller disk SEAGATE ST118202LC key 1
peripheral disk unit 0
adapter SCSI QLISP1040 key 1
controller audio RAD Audio Processor key 0
controller display SGI-SI key 0
[...]
A few days later, I had been able to get a kernel to run on the Octane and try
its chance at identifying devices.
Date: Wed, 20 Feb 2008 22:32:54 +0000
From: Miod Vallat
To: Jasper Lievisse Adriaanse, Joel Sing, Martin Reindl
Subject: Octane status and diff
I have reached the point where PCI device enumeration almost works
(PCI configuration space access). bus_space for PCI devices is probably
completely wrong, so actual PCI drivers will not work yet.
This means that, right now, it boots into:
>> bootp()/bsd.octane
Setting $netaddr to 10.0.1.158 (from server )
Obtaining /bsd.octane from server
2900544+401224 entry: 0xa800000020020000
ARCS64 Firmware Version 64.0
memory descriptor: type 0 start 0x0 count 0x1
memory descriptor: type 1 start 0x1 count 0x1
memory descriptor: type 7 start 0x2 count 0x2
memory descriptor: type 3 start 0x20004 count 0x1c
memory descriptor: type 5 start 0x20020 count 0x327
memory descriptor: type 3 start 0x20347 count 0xbb9
memory descriptor: type 6 start 0x20f00 count 0x100
memory descriptor: type 3 start 0x21000 count 0x7000
SR=340050a0
Found SGI-IP30, setting up.
memory bank 0: 80430000
memory from 0x20000000000 to 0x28000000000
memory bank 1: 00000000
memory bank 2: 00000000
memory bank 3: 00000000
memory bank 4: 00000000
memory bank 5: 00000000
memory bank 6: 00000000
memory bank 7: 00000000
tentative console uart address @0x900000001f620178
Initial setup done, switching console.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2008 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.2-current (OCTANE) #66: Wed Feb 20 21:50:54 GMT 2008
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/OCTANE
real mem = 134217728 (128MB)
rsvd mem = 1064960 (1MB)
avail mem = 122413056 (116MB)
mainbus0 at root
cpu0 at mainbus0: MIPS R10000 CPU rev 2.7 174 MHz with R10000 FPU rev 0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 1024KB 2 way
xbow0 at mainbus0: XBow revision 4
xheart0 at xbow0 widget 8: Heart revision 4
"HQ4 / ImpactSR" revision 2 at xbow0 widget 9 not configured
xbridge0 at xbow0 widget 15: Bridge revision 3
pci0 at xbridge0 bus 0
isp0 at pci0 dev 0 function 0 "QLogic ISP1020" rev 0x05: irq not
implemented yet
isp0: invalid NVRAM header
scsibus0 at isp0: 16 targets
isp0: Polled Mailbox Command (0x15) Timeout
isp0: Polled Mailbox Command (0x15) Timeout
... and spins writing this message every few seconds.
However, if I disable isp:
>> bootp()/bsd.octane -c
[...]
OpenBSD 4.2-current (OCTANE) #66: Wed Feb 20 21:50:54 GMT 2008
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/OCTANE
real mem = 134217728 (128MB)
rsvd mem = 1064960 (1MB)
avail mem = 122413056 (116MB)
User Kernel Config
UKC> disable isp
17 isp* disabled
UKC> quit
Continuing...
mainbus0 at root
cpu0 at mainbus0: MIPS R10000 CPU rev 2.7 174 MHz with R10000 FPU rev
0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 1024KB 2 way
xbow0 at mainbus0: XBow revision 4
xheart0 at xbow0 widget 8: Heart revision 4
"HQ4 / ImpactSR" revision 2 at xbow0 widget 9 not configured
xbridge0 at xbow0 widget 15: Bridge revision 3
pci0 at xbridge0 bus 0
"QLogic ISP1020" rev 0x05 at pci0 dev 0 function 0 not configured
"QLogic ISP1020" rev 0x05 at pci0 dev 1 function 0 not configured
"SGI Rad1" rev 0xc0 at pci0 dev 3 function 0 not configured
clock0 at mainbus0 ticker on int5 using count register
/dev/ksyms: Symbol table not valid.
softraid0 at root
boot device: lookup 'sd0a' failed.
root device:
You'll find below the diff I have been using to reach this state. There
are ugly cheats to get com to get compiled in (so that we can use the
com console routines although we can't attach it yet because we lack an
IOC3 driver at the moment.
Anyway, here are my plans with this code:
- I plan to put a large part of the interrupt knowledge into the xheart
(for ip30) and xhub (for ip27) drivers, which are empty shells for
now. There will be various function pointers which will be filled by
either xheart or xhub as it attaches.
- I need to work on the clock attachment since right now it's @mainbus
because there is no other place to put it. But this needs a basic IOC3
driver first.
- I need to complete the pci bridge code. You may notice in the previous
dmesg that the IOC3 at pci0 dev 2 has not been found at all, and that
the isp driver went nuts quickly.
- I need to find out a better way to find out the serial console
address. Joel found its address but we do not understand (yet) why it
is there.
The short term work will be mainly on xbridge.c and xbridgevar.h, if
only to describe more registers and initialize them. And I'll need to
give the ip27 a try this weekend (I borrowed an O200 from Matthieu's
lab today).
My plan is still to get as much as possible done by sunday, since
afterwards I'll concentrate more on release tasks. I don't think this
code will make 4.3 anyway...
Miod
PS: The kernel address issue. This can really be fixed at the
bootloader. Right now we start our kernel at a location low in physical
memory, making sure there is room for the ARCBios reserved pages in low
memory and the bootloader code itself. What can be done is linking the
kernel at an XKPHYS address, and have the bootloader handle this
correctly. Right now the bootloader truncates the load address to 32
bits, which is why it needs to be in KSEG, for sign-extension to provide
a correct 64 bit pointer. By fixing it to use the real 64 bit address,
the existing load address converted to XKPHYS should work on O2, O200
and Octane - I need to check the CCA space we use on r10k is ok on r5k
though.
Another possibility is to stick to the KSEG address, but then we'll need
to provide several boot loaders because the Octane prom will not load a
kernel in KSEG. And bsd.rd would need to be linked to an XKPHYS address
too... Oh well, it will need some tinkering.
And don't worry about installation scripts, they can decide which
bootloader to use from this:
sysctl hw.model | sed 's,.*(\(IP.*\)),\1,g'
PPS: rebooting and powering off from ddb work nicely.
[...]
Joel Sing started working on a driver for the IOC3 chip, and shared code on the
23th. I sent him back a new diff with both our work-in-progress changes merged.
Date: Sat, 23 Feb 2008 15:22:29 +0000 From: Miod Vallat To: Joel Sing Subject: Re: Introducing ioc(4) Current code with ioc changes, com@ioc, and new dsrtc attachment. I'm losing hair on the onewire stuff at the moment. [...]
The onewire comment is related to the fact that, on Octane and, to a lesser
extent on Origin 200, some system information, mainly serial numbers as well as
the Ethernet address of the onboard interface, were stored on iButton
Dallas chips,
which were small write-once memory devices (i.e. it was possible
to append data, for example to register a system component upgrade, but not to
erase existing data) connected to a
1-Wire bus.
Thankfully there was already support for some 1-Wire devices under OpenBSD
thanks to the work of Alexander Yurchenko, so
all I had to do in order to be able to get the Ethernet address was to
understand how to control the 1-Wire controller found in the Octane.
Between frustrating kernel tests on the Octane, I started tinkering with an
Origin 200 system.
It turns out these systems have a stripped-down ARCS firmware
with only enough data and function pointers to be able to load a bootloader
and run it, and not much else. All the system configuration data is set up in
a tree of complex structures in memory, laid out by the PROM during its
initialization, and called the KLCONFIG. I don’t know what the letters
KL are supposed to mean, but until I got familiar with it, it sounded
like KLingon to me.
There is a short description of these structures in IRIX
/usr/include/sys/SN/klconfig.h header file.
/*
* The KLCONFIG structures store info about the various BOARDs found
* during Hardware Discovery. In addition, it stores info about the
* components found on the BOARDs.
*/
[...]
/*
* The KLCONFIG area is organized as a LINKED LIST of BOARDs. A BOARD
* can be either 'LOCAL' or 'REMOTE'. LOCAL means it is attached to
* the LOCAL/current NODE. REMOTE means it is attached to a different
* node.(TBD - Need a way to treat ROUTER boards.)
*
* There are 2 different structures to represent these boards -
* lboard - Local board, rboard - remote board. These 2 structures
* can be arbitrarily mixed in the LINKED LIST of BOARDs. (Refer
* Figure below). The first byte of the rboard or lboard structure
* is used to find out its type - no unions are used.
* If it is a lboard, then the config info of this board will be found
* on the local node. (LOCAL NODE BASE + offset value gives pointer to
* the structure.
* If it is a rboard, the local structure contains the node number
* and the offset of the beginning of the LINKED LIST on the remote node.
* The details of the hardware on a remote node can be built locally,
* if required, by reading the LINKED LIST on the remote node and
* ignoring all the rboards on that node.
*
* The local node uses the REMOTE NODE NUMBER + OFFSET to point to the
* First board info on the remote node. The remote node list is
* traversed as the local list, using the REMOTE BASE ADDRESS and not
* the local base address and ignoring all rboard values.
*
*
KLCONFIG
+------------+ +------------+ +------------+ +------------+
| lboard | +-->| lboard | +-->| rboard | +-->| lboard |
+------------+ | +------------+ | +------------+ | +------------+
| board info | | | board info | | |errinfo,bptr| | | board info |
+------------+ | +------------+ | +------------+ | +------------+
| offset |--+ | offset |--+ | offset |--+ |offset=NULL |
+------------+ +------------+ +------------+ +------------+
+------------+
| board info |
+------------+ +--------------------------------+
| compt 1 |------>| type, rev, diaginfo, size ... | (CPU)
+------------+ +--------------------------------+
| compt 2 |--+
+------------+ | +--------------------------------+
| ... | +--->| type, rev, diaginfo, size ... | (MEM_BANK)
+------------+ +--------------------------------+
| errinfo |--+
+------------+ | +--------------------------------+
+--->|r/l brd errinfo,compt err flags |
+--------------------------------+
*
* Each BOARD consists of COMPONENTs and the BOARD structure has
* pointers (offsets) to its COMPONENT structure.
* The COMPONENT structure has version info, size and speed info, revision,
* error info and the NIC info. This structure can accomodate any
* BOARD with arbitrary COMPONENT composition.
*
* The ERRORINFO part of each BOARD has error information
* that describes errors about the BOARD itself. It also has flags to
* indicate the COMPONENT(s) on the board that have errors. The error
* information specific to the COMPONENT is present in the respective
* COMPONENT structure.
*
* The ERRORINFO structure is also treated like a COMPONENT, ie. the
* BOARD has pointers(offset) to the ERRORINFO structure. The rboard
* structure also has a pointer to the ERRORINFO structure. This is
* the place to store ERRORINFO about a REMOTE NODE, if the HUB on
* that NODE is not working or if the REMOTE MEMORY is BAD. In cases where
* only the CPU of the REMOTE NODE is disabled, the ERRORINFO pointer can
* be a NODE NUMBER, REMOTE OFFSET combination, pointing to error info
* which is present on the REMOTE NODE.(TBD)
* REMOTE ERRINFO can be stored on any of the nearest nodes
* or on all the nearest nodes.(TBD)
* Like BOARD structures, REMOTE ERRINFO structures can be built locally
* using the rboard errinfo pointer.
[...]
*/
Date: Wed, 27 Feb 2008 05:59:48 +0000
From: Miod Vallat
To: Jasper Lievisse Adriaanse
Cc: Martin Reindl, Joel Sing
Subject: Re: O2/O200/Octane: we'll need three kernels
[...]
I have fixed a few things and I've been able to panic in uvm bowels
shortly after the copyright message:
>> bootp()/bsd.ip27
Setting $netaddr to 10.0.1.145 (from server )
Obtaining bsd.ip27 from server
2908192+400344 entry: 0xa800000000020000
ARCS64 Firmware Version 64.0
memory descriptor: type 0 start 0x0 count 0x1
memory descriptor: type 1 start 0x1 count 0x1
memory descriptor: type 3 start 0x19 count 0x7
memory descriptor: type 5 start 0x20 count 0x328
memory descriptor: type 3 start 0x348 count 0xfb8
memory descriptor: type 6 start 0x1300 count 0x100
memory descriptor: type 3 start 0x1400 count 0x100
memory descriptor: type 6 start 0x1500 count 0x300
memory descriptor: type 6 start 0x1800 count 0x200
memory descriptor: type 7 start 0x1a00 count 0x600
SR=246000a0
CFG=6c0aa985
Found SGI-IP27, setting up.
config @0x9600000000004000
magic 0xbeedbabe version 0
console 0x9200000008620178 baud 9600
Machine is in M mode.
Region present 0xffffffffffffffff.
Calias size 0x2.
board type 11 components 4
cpu type 934 225Mhz cache 2MB speed 225Mhz
cpu type 934 225Mhz cache 2MB speed 225Mhz
hub port -1 speed 90MHz
memory 1024MB, select 0
bank 1 256MB
bank 2 512MB
bank 3 128MB
bank 4 128MB
board type 41 components 0
component type 26
board type 21 components 4
component type 5
component type 11
component type 11
component type 6
sys_config.console_io.bus_base = 0x9200000008000000
sys_config.cons_ioaddr = 0x620178
kernel: 0x20-0x348
page 0: 0x19-0x20
uvm_page_physload(0x19,0x20)
page 1: 0x348-0x1300
uvm_page_physload(0x348,0x1300)
page 2: 0x1400-0x1500
uvm_page_physload(0x1400,0x1500)
page 3: 0x2000-0x40000
uvm_page_physload(0x2000,0x40000)
Initial setup done, switching console.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2008 OpenBSD. All rights reserved. http://www.OpenBSD.org
Trap cause = 7 Frame 0xffffffff8001edb8
Trap PC 0xa8000000001ffb3c RA 0xa8000000001ffb20 fault 0xffffffffc93c9a70
0xa8000000001ffb3c ra 0xa8000000001ffb20 sp 0xffffffff8001ef10 (0xffffffffca311000,0x100027,0x1000,0xffffffff8001ee64)
0xa8000000001ffb20 ra 0x0 sp 0xffffffff8001ef10
User-level: pid 0
stopped on non ddb fault
Stopped at 0xa8000000001ffb3c: sd v1,0(v0)
0xa8000000001ff9d8 (ffffffffca311000,100027,1000,ffffffff8001ee64) sp ffffffff8001ef10 ra a8000000001ffb20, sz 0
0xa8000000001ff9d8 (ffffffffca311000,100027,1000,ffffffff8001ee64) sp ffffffff8001ef10 ra 0, sz 0
User-level: pid 0
ddb>
That's progress! Although not much.
The 1-Wire side quest completed, and it was time to address interrupt handling.
Date: Thu, 28 Feb 2008 20:04:00 +0000
From: Miod Vallat
To: Jasper Lievisse Adriaanse, Joel Sing, Martin Reindl
Subject: octane news
Bah. I'm busy working on my employer's answer to an European Space
Agency 18 months contract. Not much left to hack...
* Octane
- there's a 1-Wire controller on the HEART widget, it gives us the
mobo serial number.
- reworked the 1-Wire stuff to share code, split the EEPROM driver
into the Ethernet Address flavor and the serial number flavor, got
rid of the intermediate iocow layer. Also, onewire attaches early
so that (eventually) other devices can access the information they
provide.
- no work on the interrupt stuff )-:
- no improvement on the isp side although I've experimented a lot of
things (this is why I want minimal ip27 support so that I can test
other pci devices, as I don't have the pci cardcage in ``my''
Octane).
* Origin 200
- fixed memory segment initialization: memory below 32MB gets filled
by ARCBios with a bug workaround, memory above 32MB gets filled from
the node information. This lets me past copyright.
- unfortunately uvm_km_page_init() dies with a bus error, which I am
currently investigating.
* Short-term plans
- interrupt handling on IP30 (necessary so that ioc subdevices can use
interrupts, to be confirmed with a working bsd.rd). This will allow
Joel to work on the Ethernet driver.
- Find out why IP27 is dying so early and fix it.
No up-to-date diff yet, although I have uploaded recent kernels. Here's
what I now get on Octane:
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2008 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.3-beta (GENERIC-IP30) #45: Thu Feb 28 19:45:09 GMT 2008
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/GENERIC-IP30
real mem = 134217728 (128MB)
rsvd mem = 1064960 (1MB)
avail mem = 122408960 (116MB)
mainbus0 at root
cpu0 at mainbus0: MIPS R10000 CPU rev 2.7 174 MHz with R10000 FPU rev 0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 1024KB 2 way
clock0 at mainbus0: ticker on int5 using count register
xbow0 at mainbus0: XBow revision 4
xheart0 at xbow0 widget 8: Heart revision 4
onewire0 at xheart0
owserial0 at onewire0 "16kb EPROM" sn 00000012fd21
owserial0: "PM20175MHZ" serial 030-1208-002
"ImpactSR" revision 2 at xbow0 widget 9 not configured
xbridge0 at xbow0 widget 15: Bridge revision 3
pci0 at xbridge0 bus 0
isp0 at pci0 dev 0 function 0 "QLogic ISP1020" rev 0x05: irq 8
isp0: Polled Mailbox Command (0x8) Timeout
isp0: Polled Mailbox Command (0x0) Timeout
isp1 at pci0 dev 1 function 0 "QLogic ISP1020" rev 0x05: irq 9
isp1: Polled Mailbox Command (0x8) Timeout
isp1: Polled Mailbox Command (0x0) Timeout
ioc0 at pci0 dev 2 function 0 "SGI IOC3" rev 0x01
onewire1 at ioc0
owmac0 at onewire1 "1kb EPROM" sn 0000013f1552
owmac0: Ethernet Address 08:00:69:0b:f9:a1
owserial1 at onewire1 "16kb EPROM" sn 00000012034b
owserial1: "PWR.SPPLY.SR" serial 060-0028-003
owserial2 at onewire1 "16kb EPROM" sn 0000001e4edf
owserial2: "FP1" serial 030-0891-003
com0 at ioc0 base 0x00020178: ns16550a, 16 byte fifo
com0: console
com1 at ioc0 base 0x00020170: ns16550a, 16 byte fifo
dsrtc0 at ioc0: DS1687
"SGI Rad1" rev 0xc0 at pci0 dev 3 function 0 not configured
/dev/ksyms: Symbol table not valid.
softraid0 at root
boot device: lookup 'sd0a' failed.
root device:
That Octane work was stalled due to the OpenBSD 4.3 release process, during
which I got to be the person building the OpenBSD/sgi release binaries.
I intended to resume working on it, but kept being distracted by other work.
I commited the work-in-progress code to the OpenBSD CVS repository on april 7th,
in order to make it public, so that other people could tinker with it if they
would like to.
Late april, I received in private mail a report that an O2 system
where OpenBSD would report more memory than its PROM.
Thankfully the reporter agreed to test a kernel with extra debug information,
which allowed me to figure out the cause of the problem.
A fix was quickly devised.
Date: Tue, 29 Apr 2008 17:55:13 +0000
From: Miod Vallat
To: private OpenBSD mailinglist
Subject: better SGI O2 memory detection
I have received a report in private mail of bsd not reporting memory
correctly on an O2.
hinv would report 576MB:
Memory size: 576 Mbytes
and the kernel would report this:
real mem = 671088640 (640MB)
rsvd mem = 7020544 (6MB)
avail mem = 877572096 (836MB)
While having more available memory than existing is an impressive
achievement, there is no doubt the machine will misbehave when trying to
use memory which is not there.
The discrepency [sic] between real and available memory hinted that physmem
would not increase when memory banks were counted, which in turns hints
at overlapping banks.
So I experimented with many weird memory layouts in an O2 and some debug
information, and this was exactly what I saw. The logic in
crime_configure_memory() assumes empty banks are reported as address
zero.
This is wrong!!!
What really happens is that CRIME reports an overlapping memory
region - but not always the same!
Here's an example. The machine was populated with 2x32MB, then 4x64MB,
then 2x32MB, for a total of 384MB memory. CRIME happily reports:
bank 0 32MB at 256MB
bank 1 32MB at 288MB
bank 2 128MB at 0MB
bank 3 32MB at 256MB -- empty due to 2x64MB merge
bank 4 128MB at 128MB
bank 5 32MB at 256MB -- empty due to 2x64MB merge
bank 6 32MB at 320MB
bank 7 32MB at 352MB
An unmodified GENERIC kernel would not even boot with this setup - it
would die early in uvm initialization.
In this case the empty slots simply mimic the first one (32MB at 256MB).
But the value may change. Here is a setup with 2x32+2x64+2x32:
bank 0 32MB at 128MB
bank 1 32MB at 160MB
bank 2 128MB at 0MB
bank 3 32MB at 128MB -- empty due to 2x64MB merge
bank 4 32MB at 192MB
bank 5 32MB at 224MB
bank 6 32MB at 128MB -- empty
bank 7 32MB at 128MB -- empty
Now let's remove all 64MB DIMMs and keep 4x32MB:
bank 0 32MB at 0MB
bank 1 32MB at 32MB
bank 2 32MB at 64MB
bank 3 32MB at 96MB
bank 4 32MB at 0MB -- empty
bank 5 32MB at 0MB -- empty
bank 6 32MB at 0MB -- empty
bank 7 32MB at 0MB -- empty
And finally, for more fun, 2x64+4x32:
bank 0 128MB at 0MB
bank 1 128MB at 0MB -- empty due to 2x64MB merge
bank 2 32MB at 128MB
bank 3 32MB at 160MB
bank 4 32MB at 192MB
bank 5 32MB at 224MB
bank 6 128MB at 0MB -- empty
bank 7 128MB at 0MB -- empty
So basically:
- every time you'll put a pair of 64MB DIMMs, CRIME will merge them as a
single 128MB bank, and the bank following it is empty.
- really empty banks (such as #6 and #7 in the last example) copy the
value of the first bank.
If you follow SGI recommandation [sic] to put the larger DIMMs in the lower
banks, it will configure them at the lowest address (below 256MB), so
the copies of bank0 will be ignored. But since the machine won't mind
booting with the DIMMs in random order, we have to cope with this
nevertheless.
With this in mind, I cooked the following diff.
Please test this on your O2s and let me know if the kernel still reports
as much memory as hinv, and available memory lower [than] real memory.
Miod
After about a week of testing, this fix hit the tree on may 4th.
In late july, I borrowed a PCI WaveLan card from a friend. That card
was in fact a PCI/PCMCIA bridge with a PCMCIA WaveLan card plugged into it.
wi0 at pci0 dev 3 function 0 "US Robotics WL11000P" rev 0x02: irq 11 wi0: "Lucent Technologies, WaveLAN/IEEE, Version 01.01" wi0: Firmware 6.04 variant 1, address 00:60:1d:1d:21:61 on sgi. O2 or something more interesting? O2. a pci one eh. who wants the diff? I do. It needed a diff? There was stuff wrong? bus_space fixes Oh. sgi. Right :) then it worked out of the box nice. miod has pci devices? I borrow them.
Octane progress would still have to wait, however.
[go back to the previous part][skip
to the next part]
2009, NetBSD
The machine coverage of NetBSD on sgi
increased
to reach two of the oldest
MIPS-based SGI machines, the Personal Iris 4D systems (IP6 and IP10), again
thanks to the hard work of Steve Rumble.
Enable Personal IRIS 4D/20 and 4D/25 support:
- Adapt int(4) to handle the INT1 chip
- Move generic rtc clocks out of hpc/ and into dev/
- Handle the very strangely wired eeprom and other bits in arcemu
- Sprinkle MACH_SGI_IP6 as necessary
- Enable IP6/IP10 devices in GENERIC32_IP12. Yes, the naming is poor but
there's no winning with kernel/hw compatibility on sgimips...
Tested on my 4D/25. Doesn't (appear to) break macallan@'s IP22.
A few days later, Michael Lorenz contributed X server drivers for both
the XL (newport) graphics for Indy, and the O2 frame buffer.
2009, OpenBSD
At about the same time work occurred on the O2 display in NetBSD, Joel
Sing was also working on the OpenBSD side and added
console
acceleration
on february 24th.
Add support for hardware acceleration to gbe(4). This provides an accelerated framebuffer for the console on SGI O2 workstations. X is still supported via wsfb(4) by switching back to the unaccelerated linear framebuffer mode. Some hardware details and magic numbers from NetBSD's crmfb(4) driver.
In april, prompted by a failed experiment by Joel Sing to get Octane interrupts
working, I resumed work on this machine.
Date: Sun, 12 Apr 2009 18:25:18 +0000
From: Miod Vallat
To: Jasper Lievisse Adriaanse, Joel Sing , Martin Reindl
Cc: Theo de Raadt
Subject: Octane status
I have commited all bits necessary to get RAMDISK kernels to run on
ip30 (except for the kernel configuration files).
It's a good thing I got myself a second Octane, because the new one has
an R12000 cpu, a version 4 Xbridge (thus allowing me to test the >= 4
codepath), and exposed bugs the other one did not.
So after enough tinkering, I got both machines to run. There are kluges
all over ioc.c though.
I have changed the locators of ioc subdevices, to not use `irq' as the
second locator - these are not really irq but subdevice numbers, so the
config stanzas become:
com0 at ioc? base 0x00020178 dev 1
com1 at ioc? base 0x00020170 dev 2
I am considering removing the ioc locators from the kernel configuration
file and have ioc provide them itself from a table (kinda like obio on
many other architectures) to prevent people from playing with locators.
But that's not important at this point.
Now, the todolist is:
- in parallel:
- write an onboard Ethernet driver. I think Joel can be bribed to
work on this (-:
- get o200 to run further (I need to fix the widget enumeration logic,
which is wrong on this machine, and then write the pci xhub code).
That's what I'll be working on ASAP.
- once we have a machine with Ethernet (either Octane onboard or o200
with an extra PCI card), introduce distinct kernel configuration files
(GENERIC-IP*, RAMDISK-IP*), and discuss this to explain why this
is necessary. Update the distrib/ machinery to use *-IP32 for O2, do
not build Octane media yet.
- similar changes to the disk bootblocks: they will need different link
address. Devise a proper machinery to get the bootblocks with
different names in /usr/mdec, alter installation media to use ip32
bootloader.
- probably add a sysctl to the kernel to report what kind of boot and
kernel it needs (IP27/IP30/IP32). Start using this in the installation
media, with a small sysctl retrieval tool a la ``disknames''. Or maybe
a proper regexp on the kernel name in dmesg would be enough (since it
will be RAMDISK-IP##).
- once we have installation media able to work on O2, Octane (and
probably Origin 200 by this time), celebrate, then start working on
video and glass console... and on SMP... etc, etc. We're not there
yet.
[...]
The next day, I also had made some progress on Origin 200, which was however
behind.
From: Miod Vallat
To: Jasper Lievisse Adriaanse
Cc: Joel Sing, Martin Reindl, Theo de Raadt
Subject: Re: Octane status
[...]
> > - get o200 to run further (I need to fix the widget enumeration logic,
> > which is wrong on this machine, and then write the pci xhub code).
> > That's what I'll be working on ASAP.
> I'll happily test diffs on my o200's.
Well I'm slowly getting there, here is what I got minutes ago:
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.5-current (RAMDISK-IP27) #10: Mon Apr 13 13:21:34 GMT 2009
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/RAMDISK-IP27
real mem = 1073741824 (1024MB)
rsvd mem = 12591104 (12MB)
avail mem = 1002553344 (956MB)
mainbus0 at root
cpu0 at mainbus0: MIPS R10000 CPU rev 3.4 225 MHz with R10000 FPU rev 0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 2048KB 2 way
clock0 at mainbus0: ticker on int5 using count register
node0 at mainbus0
mboard nasid 0 slot 1 not configured
mplane nasid 0 slot 48 not configured
xbridge0 at node0 nasid 0 slot 1 revision 4
pci0 at xbridge0 bus 0
isp0 at pci0 dev 0 function 0 "QLogic ISP1020" rev 0x05: couldn't
establish interrupt at irq 0
isp1 at pci0 dev 1 function 0 "QLogic ISP1020" rev 0x05: couldn't
establish interrupt at irq 1
ioc0 at pci0 dev 2 function 0 "SGI IOC3" rev 0x01: failed to establish
superio interrupt!
rd0: fixed, 8192 blocks
boot device: lookup 'sd0a' failed.
root on rd0a swap on rd0b dump on rd0b
However, I have needed to change many things in sgi/xbow/ (basically to
extend the code from an ip30 view of the world, with a single set of
widgets and a friendly xbow arbiter, to an ip27/ip35 view of the world,
with multiple widget locations and different device tree anchorage, see
the ``node'' bus in the dmesg above).
I need to work on interrupt support for ip27 and hopefully I'll reach
single-user mode soon. Then I'll commit the o200 parts.
In the meantime, OpenBSD users had noticed the Octane activity and
asked for more
information.
Date: Wed, 15 Apr 2009 00:44:52 +0000 From: Naoki Hamada To: openbsd-sgi Subject: octane support Hello guys, I recently got an octane and want to make it a bsd machine. I am very happy to find there are commits around IP30. I tried to generate a kernel but there lack some IP30 parts (config file, for example) yet. If they are commited, I would willingly test them and hopefully improve them. Naoki Hamada
To which I
replied:
Date: Wed, 15 Apr 2009 18:59:21 +0000
From: Miod Vallat
To: Naoki Hamada
Cc: sgi@openbsd.org
Subject: Re: octane support
> I am very happy to find there are commits around IP30. I tried to
> generate a kernel but there lack some IP30 parts (config file, for
> example) yet. If they are commited, I would willingly test them
> and hopefully improve them.
The lack of the kernel configuration files is intentional (-: They will
appear when the machines are a bit more usable.
At the moment, Octane will run using a serial console, and the onboard
SCSI controllers. However, the onboard interface is not supported yet,
and should you be the happy owner of a PCI Cardcage extension, or a PCI
Shoehorn XIO card, their PCI devices will not configure correctly yet.
As long as the machine can not talk over the network, running BSD on it
is quite useless.
Please give us a bit more time to make progress on this!
Miod
PS: and before people ask, here is the log of bsd.rd booting on one of
my Octanes:
Running power-on diagnostics...
System Maintenance Menu
1) Start System
2) Install System Software
3) Run Diagnostics
4) Recover System
5) Enter Command Monitor
Option? 5
Command Monitor. Type "exit" to return to the menu.
>> bootp()/bsd.ip30
Setting $netaddr to 10.0.1.211 (from server )
Obtaining /bsd.ip30 from server
5859248+484952 entry: 0xa800000020020000
ARCS64 Firmware Version 64.0
Found SGI-IP30, setting up.
Initial setup done, switching console.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.5-current (RAMDISK-IP30) #75: Wed Apr 15 18:24:43 GMT 2009
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/RAMDISK-IP30
real mem = 1073741824 (1024MB)
rsvd mem = 1064960 (1MB)
avail mem = 1013821440 (966MB)
mainbus0 at root
cpu0 at mainbus0: MIPS R12000 CPU rev 2.3 300 MHz with R10000 FPU rev 0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 2048KB 2 way
clock0 at mainbus0: ticker on int5 using count register
xbow0 at mainbus0: XBow revision 4
xheart0 at xbow0 widget 8: Heart revision 6
onewire0 at xheart0
owserial0 at onewire0 "16kb EPROM" sn 0000003c3d84
owserial0: "PM10300MHZ" serial 030-1355-001
"ImpactSR" revision 2 at xbow0 widget 12 not configured
xbridge0 at xbow0 widget 15: Bridge revision 4
pci0 at xbridge0 bus 0
isp0 at pci0 dev 0 function 0 "QLogic ISP1020" rev 0x05: irq 0
isp0: invalid NVRAM header
scsibus0 at isp0: 16 targets, initiator 7
sd0 at scsibus0 targ 1 lun 0: SCSI3 0/direct
+fixed
sd0: 8683MB, 512 bytes/sec, 17783249 sec total
isp1 at pci0 dev 1 function 0 "QLogic ISP1020" rev 0x05: irq 1
isp1: invalid NVRAM header
scsibus1 at isp1: 16 targets, initiator 7
ioc0 at pci0 dev 2 function 0 "SGI IOC3" rev 0x01: superio irq 4, ethernet irq 2
onewire1 at ioc0
owmac0 at onewire1 "1kb EPROM" sn 0000029e542f
owmac0: Ethernet Address 08:00:69:13:4b:f8
owserial1 at onewire1 "16kb EPROM" sn 000000348d70
owserial1: "FP1" serial 030-0891-003
owserial2 at onewire1 "16kb EPROM" sn 000000631bf1
owserial2: "PWR.SPPLY.ER" serial 060-0035-003
com0 at ioc0 base 0x00020178 dev 1: ns16550a, 16 byte fifo
com0: console
com1 at ioc0 base 0x00020170 dev 2: ns16550a, 16 byte fifo
dsrtc0 at ioc0: DS1687
"SGI Rad1" rev 0xc0 at pci0 dev 3 function 0 not configured
rd0: fixed, 8192 blocks
boot device: sd0
root on rd0a swap on rd0b dump on rd0b
WARNING: clock time much less than file system time
WARNING: using file system time
WARNING: CHECK AND RESET THE DATE!
erase ^?, werase ^W, kill ^U, intr ^C, status ^T
(I)nstall, (U)pgrade or (S)hell? S
# ls
.profile etc install.sub sbin usr
bin install mnt tmp var
dev install.md mnt2 upgrade
#
You might remember the friend I had mentioned got two Fuel systems in march
2007. As a joke, he had dared me several times to make OpenBSD run on them.
As work on the Origin 200 and the Octane was slowly making progress, I
would have liked to also test it on the IP35 family, in order to try to
understand the differences between IP27 and IP35, and make sure the way I would
design or write my code to support IP27 would not get in the way of a possible
future IP35 support. I asked him whether he would accept to lend me one of his
Fuel systems, and he gladly did (telling me “don’t you dare to give it back
until you have a working OpenBSD port”.)
Date: Wed, 15 Apr 2009 20:18:54 +0000 From: Miod Vallat To: Jean-Marc Harang Subject: Fuel Quand est-ce que je peux passer t'emprunter une Fuel ? Pas besoin de disque dedans. [When can I drop by and borrow a Fuel? I don't need a disk in it.] C'est un peu tôt compte tenu de l'état encore rustique du support Octane et Origin 200, mais en fait jouer avec maintenant me permettrait d'avoir plus d'informations pour effectuer des choix de conception (et en particulier dans quelle mesure Fuel est une Origin 200 dans une configuration particulière, où s'il y a suffisamment de différences pour justifier une vie séparée). [It is a bit early given the current rustic state of the Octane and Origin 200 support, but being able to tinker with it now would let me gather more information in order to make good design choices (and especially figure out how much resemblance there is between Fuel and Origin 200, and if there are enough differences to warrant separate code bases.)]
Looking at the Linux code, the Octane and O2 onboard Ethernet controllers,
albeit different, were very similar on many aspects. I tried to adapt the
existing OpenBSD O2 Ethernet driver into running on the Octane, but it did not
work yet. Not knowing whether I would spend more time on it, as there were so
many other things to work on and I could use PCI Ethernet cards in the Origin
200, I shared the code with Joel Sing and Jasper Lievisse Adriaanse in
case they would have time to help.
Date: Thu, 16 Apr 2009 20:06:31 +0000 From: Miod Vallat To: Joel Sing, Jasper Lievisse Adriaanse Subject: Octane Ethernet boilerplate I have had a look at the linux code for this beast, and came up with this. There are some similarities with if_mec on O2, but MEC is probably a later design correcting a few annoying problems in EF (such as having to get TX interrupts after each TX ring because the TX errors are reported as interrupts and not as per-descriptor status information). So the following code is if_mec with s/mec/ef/ and updated to play with the EF registers. However, don't get your hopes up: this does not work. What works: - ifconfig. - mii access (but this is trivial). What doesn't work: - everything else. Actually, I can't get the chip to interrupt. I have added a dump of the interrupt registers in the watchdog timeout, and it always remain stuck at zero. Things to consider: I did not port the linux code which looks for ssram and sets bits in the MCR register. This probably matters, and the SSRAM might be used as internal work memory, I don't know. So this is one of the first things to try. Then the ring buffer managment might be wrong as well. But this should provide a good starting point for anyone wanting to get this further. All the tedious task of writing the register defines and the function stubs is done (-: Miod PS: Configuration stanzas: ef* at ioc? icsphy* at mii? # that's what I have here nsphy* at mii? # that's what the Linux code suggests ukphy* at mii? PPS: I'll probably commit the ioc.c change soon, it cleans things up and is not intrusive. [...]
I spent the next few days trying to boot a kernel on the Fuel. The first few
tries failed miserably, because the address my kernel was linked at was too low
in memory and would overlap the ARCS reserved area (which is twice larger on
IP35 systems than on IP27 systems.) After some trial-and-error I eventually
found an address which would let ARCS accept my kernel image (I was attempting
to boot directly from the ARCS firmware, over the network, so there were no
debug functionalities available at this state until I could have it load my
own code.)
The first successful kernel load occurred on april 19th.
It was mentioned on my next status report on the 20th.
Date: Mon, 20 Apr 2009 20:15:22 +0000
From: Miod Vallat
To: Jasper Lievisse Adriaanse, Joel Sing, Martin Reindl
Cc: Theo de Raadt
Subject: latest Octane status
Ok, here is the current status report of IP27/IP30/IP35 support:
* IP30 (Octane)
- jsing@ and I are noticing that isp(4) freaks out eventually, after
enough I/O, and preferrably under load. I am suspecting a bug in the
interrupt code which would cause reentrancy or spl not being honored
correctly.
- jsing@ is working on the on-board Ethernet driver; he's been able to
get interrupts from it, which is much better than my efforts so far,
so I'm not currently working in that area.
* IP27 (Origin 200/2000, Onyx 2)
- isp(4) does not work because it would need to do 64 bit DMA, which the
chip might not even be capable of; I have started to work on the iommu
part of the PCI bridge, which would allow isp(4) to work as badly as
on Octane (and perhaps better due to different interrupt code).
- devices plugged in the PCI slots are correctly recognized, but don't
have any resources assigned to them, as the PROM won't initialize
them; I need to work on this and see if this can converge with
kettenis@ plans.
* IP35 (Fuel, Tezro, Origin 300/3000, Onyx 300/350/4)
- I have been able to borrow a 600MHz R14000 model from a friend for a
while; after I figured out what address to link the kernel at, I've
been able to play a bit. It will be possible to use the same kernel
for IP27 and IP35, and the kernel is able to tell them apart (kind of
a kluge but reliable).
- The main difference between IP27 and IP35, apart from faster
components, is that the cpu board logic has been extended to support
four processors instead of two. Oh, and it has an USB controller
onboard too (-:
- The isp chip in this one is a high-end one (12160) and does not seem
to work yet, could be either a DMA mapping bug or lack of decent
support for this flavour in the driver. I suspect the former, and the
work on iommu will answer this question.
- No surprise with the PCI slots, their resources aren't mapped either,
so it needs the same work as IP27.
- But the major hurdle (which is puzzling the Linux guys too) is that we
don't get interrupts. I believe there is a register which needs an
explicit write to unlock remote writes to the node interrupt register.
And of course there is also the possibility of an incorrect
computation of that register address as seen from the bus, but I plan
to sort it out soon by trying the ~ 2**24 valid combinations with the
high resolution timer interrupt (-:
So, this still needs lots of work before one of these machines is
usable. But we're slowly making progress there.
Miod
PS: bootlog on the Fuel, to tease you guys:
Command Monitor. Type "exit" to return to the menu.
>> bootp()/bsd.ip27
Setting $netaddr to 10.0.1.212 (from server )
Obtaining bsd.ip27 from server
6103616+485152 entry: 0xa800000000040000
ARCS64 Firmware Version 64.0
Found SGI-IP35, setting up.
config @0x9600000000030000
magic 0xbeedbabe version 0
console 0x920000000f820178 baud 9600
Machine is in M mode.
Region present 0x1.
Calias size 0x2.
board type 11 slot 0 nasid 0 nic 0x553a318d components 4
hub widget 0 port -1 flag 0 speed 200MHz
memory 1024MB, select 0
bank 1 256MB
bank 2 256MB
bank 3 256MB
bank 4 256MB
memory from 0x0 to 0x4000000 (64 MB)
memory from 0x8000000 to 0xc000000 (64 MB)
memory from 0x10000000 to 0x14000000 (64 MB)
memory from 0x18000000 to 0x1c000000 (64 MB)
memory from 0x20000000 to 0x24000000 (64 MB)
memory from 0x28000000 to 0x2c000000 (64 MB)
memory from 0x30000000 to 0x34000000 (64 MB)
memory from 0x38000000 to 0x3c000000 (64 MB)
memory from 0x40000000 to 0x44000000 (64 MB)
memory from 0x48000000 to 0x4c000000 (64 MB)
memory from 0x50000000 to 0x54000000 (64 MB)
memory from 0x58000000 to 0x5c000000 (64 MB)
memory from 0x60000000 to 0x64000000 (64 MB)
memory from 0x68000000 to 0x6c000000 (64 MB)
memory from 0x70000000 to 0x74000000 (64 MB)
memory from 0x78000000 to 0x7c000000 (64 MB)
component widget 0 type 17
cpu type f24 600Mhz cache 4MB speed 300Mhz
board type 42 slot 0 nasid 0 nic 0xffffffffffffffff components 1
component widget 0 type 4
board type 54 slot d nasid 0 nic 0xffffffffffffffff components 1
component widget 13 type 10
board type 71 slot e nasid 0 nic 0x553a318d components 2
component widget 14 type 5
component widget 14 type 0
board type 71 slot f nasid 0 nic 0x553a318d components 4
component widget 15 type 5
component widget 15 type 37
component widget 15 type 6
component widget 15 type 34
Initial setup done, switching console.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.5-current (RAMDISK-IP27) #123: Mon Apr 20 20:09:27 GMT 2009
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/RAMDISK-IP27
real mem = 1073741824 (1024MB)
rsvd mem = 47194112 (45MB)
avail mem = 1001119744 (954MB)
mainbus0 at root
cpu0 at mainbus0: MIPS R14000 CPU rev 2.4 600 MHz with Unknown FPU type (0x94) rev 7.5
cpu0: cache L1-I 32KB D 32KB 2 way, L2 4096KB 2 way
clock0 at mainbus0: ticker on int5 using count register
xbow0 at mainbus0: XXBow revision 2
"Odyssey" revision 11 at xbow0 widget 13 not configured
xbridge0 at xbow0 widget 14: XBridge revision 2
pci0 at xbridge0 bus 0
fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x0c: can't map i/o space
xbridge1 at xbow0 widget 15: XBridge revision 2
pci1 at xbridge1 bus 0
"QLogic ISP12160" rev 0x06 at pci1 dev 1 function 0 not configured
ioc0 at pci1 dev 4 function 0 "SGI IOC3" rev 0x01
onewire0 at ioc0
ioc0: superio irq 6, ethernet irq 4
com0 at ioc0 base 0x00020178 dev 1: ns16550a, 16 byte fifo
com0: console
com1 at ioc0 base 0x00020170 dev 2: ns16550a, 16 byte fifo
ohci0 at pci1 dev 5 function 0 "Opti 82C861" rev 0x10: irq 5, version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 "Opti OHCI root hub" rev 1.00/1.00 addr 1
rd0: fixed, 8192 blocks
boot device: lookup 'sd0a' failed.
root on rd0a swap on rd0b dump on rd0b
WARNING: No TOD clock, believing file system.
WARNING: CHECK AND RESET THE DATE!
erase ^?, werase
and there it stops because the tx fifo is full and the interrupt is not
received. The fxp in pci0 is a card I put in one of the PCI slots of the
machine; I need to change the enumeration order so that the onboard
devices (i.e. pci1) attach first, BTW.
I have no idea why the FPU type is such a strange value, BTW.
Early may, I had made some progress on the Origin and Fuel front.
Date: Sun, 3 May 2009 20:27:13 +0000
From: Miod Vallat
To: Joel Sing, Jasper Lievisse Adriaanse, Theo de Raadt
Subject: Octane status
I am slowly making progress taming the hardware... here is the current
state of HEAD:
- on IP30 (Octane), no improvement, isp(4) eventually gets confused.
- on IP27 (Origin 200) and IP35 (Fuel):
- isp(4) gets better and is able to probe the busses. However there are many
error messages and non-polled I/O never completes.
- this improvement is caused by improvements in DMA operation. As a
result, USB devices now attach properly on Fuel. I have been able to
attach and detach an ukbd, which has been correctly recognized, and
a small umass device, which I could label, newfs, mount and do some
I/O on.
- configuration of PCI devices works, to some extent, and I have been
able to get an fxp(4) recognized in both machines, and able to talk
on the wire (I had to add the fxp firmware on my ramdisk filesystem,
though...)
So this means that, after I buy a large enough USB key tomorrow (the
only umass I have here is a ridiculous 128MB), I'll be able to try to
run multiuser on the Fuel. I'll also try to setup a diskless environment
on the Origin 200, although I'll have to get the fxp firmware compiled
into the kernel for this to work.
There are probably bugs lurking in the IP27 interrupt code, which will
be exposed by this setup. And once they will be fixed, I'll try to
tackle isp(4).
And now, here is the log of a session on the Fuel console, complete with
an openssl benchmark (using a statically linked openssl binary, of
course). Note the openssl results are 3 to 5 times faster than on a
300MHz RM5200 O2.
>> bootp()/bsd.ip27
Setting $netaddr to 10.0.1.212 (from server )
Obtaining bsd.ip27 from server
6106928+485152 entry: 0xa800000000040000
ARCS64 Firmware Version 64.0
Found SGI-IP35, setting up.
config @0x9600000000030000
magic 0xbeedbabe version 0
console 0x920000000f820178 baud 9600
Machine is in M mode.
Region present 0x1.
Calias size 0x2.
board type 11 slot 0 nasid 0 nic 0x553a318d components 4
hub widget 0 port -1 flag 0 speed 200MHz
memory 1024MB, select 0
bank 1 256MB
bank 2 256MB
bank 3 256MB
bank 4 256MB
memory from 0x0 to 0x4000000 (64 MB)
memory from 0x8000000 to 0xc000000 (64 MB)
memory from 0x10000000 to 0x14000000 (64 MB)
memory from 0x18000000 to 0x1c000000 (64 MB)
memory from 0x20000000 to 0x24000000 (64 MB)
memory from 0x28000000 to 0x2c000000 (64 MB)
memory from 0x30000000 to 0x34000000 (64 MB)
memory from 0x38000000 to 0x3c000000 (64 MB)
memory from 0x40000000 to 0x44000000 (64 MB)
memory from 0x48000000 to 0x4c000000 (64 MB)
memory from 0x50000000 to 0x54000000 (64 MB)
memory from 0x58000000 to 0x5c000000 (64 MB)
memory from 0x60000000 to 0x64000000 (64 MB)
memory from 0x68000000 to 0x6c000000 (64 MB)
memory from 0x70000000 to 0x74000000 (64 MB)
memory from 0x78000000 to 0x7c000000 (64 MB)
component widget 0 type 17
cpu type f24/9475 600Mhz cache 4MB speed 300Mhz
board type 42 slot 0 nasid 0 nic 0xffffffffffffffff components 1
xbow hub master link 10
widget 2 nasid 0 flg 6
widget 5 nasid 0 flg 5
widget 6 nasid 0 flg 5
widget 7 nasid 0 flg 5
board type 54 slot d nasid 0 nic 0xffffffffffffffff components 1
component widget 13 type 10
board type 71 slot e nasid 0 nic 0x553a318d components 2
component widget 14 type 5
component widget 14 type 0
board type 71 slot f nasid 0 nic 0x553a318d components 4
component widget 15 type 5
component widget 15 type 37
component widget 15 type 6
component widget 15 type 34
Initial setup done, switching console.
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.5-current (RAMDISK-IP27) #184: Sun May 3 19:32:21 GMT 2009
miod@santoire.gentiane.org:/usr/src/sys/arch/sgi/compile/RAMDISK-IP27
real mem = 1073741824 (1024MB)
rsvd mem = 47194112 (45MB)
avail mem = 1001115648 (954MB)
mainbus0 at root
cpu0 at mainbus0: MIPS R14000 CPU rev 2.4 600 MHz with Unknown FPU type (0x94) rev 7.5
cpu0: cache L1-I 32KB D 32KB 2 way, L2 4096KB 2 way
clock0 at mainbus0: ticker on int5 using count register
xbow0 at mainbus0: XXBow revision 2
"Odyssey" revision 11 at xbow0 widget 13 not configured
xbridge0 at xbow0 widget 14: XBridge revision 2
pci0 at xbridge0 bus 0
fxp0 at pci0 dev 2 function 0 "Intel 8255x" rev 0x0c, i82550: irq 2, address 00:02:b3:98:a0:99
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
xbridge1 at xbow0 widget 15: XBridge revision 2
pci1 at xbridge1 bus 0
isp0 at pci1 dev 1 function 0 "QLogic ISP12160" rev 0x06: irq 1
isp0: invalid NVRAM header
isp0: invalid NVRAM header
scsibus0 at isp0: 16 targets, initiator 7
isp0: Unhandled Response Type 0x0
isp0: Not RESPONSE in RESPONSE Queue (type 0x0) @ idx 0 (next 1) nlooked 1
sd0 at scsibus0 targ 1 lun 0: SCSI3 0/direct fixed
sd0: drive offline
scsibus1 at isp0: 16 targets, initiator 7
ioc0 at pci1 dev 4 function 0 "SGI IOC3" rev 0x01
onewire0 at ioc0
ioc0: superio irq 0, ethernet irq 4
com0 at ioc0 base 0x00020178: ns16550a, 16 byte fifo
com0: console
com1 at ioc0 base 0x00020170: ns16550a, 16 byte fifo
ohci0 at pci1 dev 5 function 0 "Opti 82C861" rev 0x10: irq 5, version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 "Opti OHCI root hub" rev 1.00/1.00 addr 1
rd0: fixed, 8192 blocks
umass0 at uhub0 port 2 configuration 1 interface 0 "NewTech Inc. USB Mass Storage Device 2.0" rev 2.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd1 at scsibus2 targ 1 lun 0: SCSI2 0/direct removable
sd1: 124MB, 512 bytes/sec, 255744 sec total
boot device: sd0
root on rd0a swap on rd0b dump on rd0b
WARNING: No TOD clock, believing file system.
WARNING: CHECK AND RESET THE DATE!
erase ^?, werase ^W, kill ^U, intr ^C, status ^T
(I)nstall, (U)pgrade or (S)hell? S
# ifconfig fxp0 10.0.1.233
# ifconfig fxp0
fxp0: flags=8843 mtu 1500
lladdr 00:02:b3:98:a0:99
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 10.0.1.233 netmask 0xff000000 broadcast 10.255.255.255
inet6 fe80::202:b3ff:fe98:a099%fxp0 prefixlen 64 scopeid 0x1
# mount -t nfs -o bg,soft,intr 10.0.1.101:/ftp /mnt2
# disklabel sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: FLASH DISK V2.0
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 15
total sectors: 255744
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0 # microseconds
track-to-track seek: 0 # microseconds
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
c: 255744 0 unused
# disklabel -E sd1
Label editor (enter '?' for help at any prompt)
> z
> a
partition: [a]
offset: [0]
size: [255744]
FS type: [4.2BSD]
> w
> q
No label changes.
# newfs -q sd1a
/dev/rsd1a: 124.9MB in 255744 sectors of 512 bytes
4 cylinder groups of 31.22MB, 1998 blocks, 4096 inodes each
# mount /dev/sd1a /mnt
# cd /mnt
# cp /mnt2/pub/OpenBSD/snapshots/sgi/
INSTALL.sgi bsd etc45.tgz misc45.tgz xfont45.tgz
MD5 bsd.rd game45.tgz xbase45.tgz xserv45.tgz
base45.tgz comp45.tgz man45.tgz xetc45.tgz xshare45.tgz
# cp /mnt2/pub/OpenBSD/snapshots/sgi/bsd .
# cd /
# umount /mnt
# mount
/dev/rd0a on / type ffs (local)
10.0.1.101:/ftp on /mnt2 type nfs (v3, udp, soft, intr, timeo=100)
# sd1 detached
scsibus2 detached
umass0 detached
#
# umount /mnt2
# mount -t nfs -o bg,soft,intr 10.0.1.101:/users /mnt
# /mnt/miod/openssl speed -elapsed
You have chosen to measure elapsed time instead of user CPU time.
To get the most accurate results, try to run this
program when this computer is idle.
Doing md2 for 3s on 16 size blocks: 75069 md2's in 3.02s
[...]
Doing 2048 bit verify dsa's for 10s: 357 2048 bit DSA verify in 10.02s
OpenSSL 0.9.8k 25 Mar 2009
built on: date not available
options:bn(64,64) md2(int) rc4(ptr,int) des(ptr,risc2,4,int) aes(partial)
+blowfish(ptr)
compiler: information not available
available timing options: USE_TOD HZ=100 [sysconf value]
timing function used: gettimeofday
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
md2 397.77k 862.59k 1206.43k 1340.22k 1384.72k
mdc2 0.00 0.00 0.00 0.00 0.00
md4 3038.31k 10572.88k 30006.93k 56058.23k 75374.34k
md5 2373.23k 8168.48k 22910.45k 41947.58k 55435.35k
hmac(md5) 2881.94k 9541.23k 25468.98k 43950.87k 56015.52k
sha1 2313.53k 7692.07k 19743.66k 32474.73k 40025.63k
rmd160 995.99k 2444.79k 4514.79k 5729.98k 6218.36k
rc4 48197.51k 55976.50k 58301.53k 58929.03k 59099.04k
des cbc 10476.48k 11178.67k 11351.42k 11395.76k 11397.73k
des ede3 3953.03k 4051.04k 4075.34k 4081.35k 4081.17k
idea cbc 0.00 0.00 0.00 0.00 0.00
seed cbc 0.00 0.00 0.00 0.00 0.00
rc2 cbc 6446.43k 6685.48k 6746.95k 6762.23k 6766.22k
rc5-32/12 cbc 0.00 0.00 0.00 0.00 0.00
blowfish cbc 17894.74k 19536.61k 20041.38k 20168.70k 20205.25k
cast cbc 12206.88k 13109.61k 13340.35k 13399.83k 13413.84k
aes-128 cbc 15667.17k 16594.19k 16867.06k 16934.30k 16950.66k
aes-192 cbc 13795.92k 14535.70k 14746.26k 14797.12k 14809.68k
aes-256 cbc 12356.77k 12931.90k 13096.85k 13137.03k 13146.48k
camellia-128 cbc 0.00 0.00 0.00 0.00 0.00
camellia-192 cbc 0.00 0.00 0.00 0.00 0.00
camellia-256 cbc 0.00 0.00 0.00 0.00 0.00
sha256 2133.24k 4972.56k 8839.23k 10993.27k 11832.71k
sha512 1908.64k 7830.54k 12894.19k 18792.53k 21679.81k
aes-128 ige 15971.20k 17650.82k 17838.77k 17841.60k 17993.71k
aes-192 ige 14203.54k 15277.27k 15463.21k 15484.90k 15675.03k
aes-256 ige 12654.50k 13516.71k 13668.84k 13736.89k 13811.78k
sign verify sign/s verify/s
rsa 512 bits 0.003300s 0.000270s 303.1 3699.4
rsa 1024 bits 0.015298s 0.000724s 65.4 1380.5
rsa 2048 bits 0.087327s 0.002358s 11.5 424.1
rsa 4096 bits 0.568740s 0.008400s 1.8 119.1
sign verify sign/s verify/s
dsa 512 bits 0.002735s 0.002964s 365.6 337.4
dsa 1024 bits 0.007250s 0.008567s 137.9 116.7
dsa 2048 bits 0.023327s 0.028059s 42.9 35.6
# umount /mnt
#
While tinkering with interrupts, I stumbled upon a bad bug in the clock
interrupt handling code.
Date: Mon, 4 May 2009 17:47:51 +0000 From: Miod Vallat To: Joel Sing, Theo de Raadt, Mark Kettenis Subject: sgi clock fix [...] Basically, the bug is so stupid noone had noticed it. We run with the clock interrupt always enabled, and each time we get a clock interrupt, we count how many integral ticks have elapsed since the last interrupt, schedule the next one, and if we are not at splclock() or splhigh(), invoke hardclock() as many times as necessary. Since the clock is implemented with a free-running counter, we have to make sure, when we schedule the next interrupt, that the counter has not already reached it. You might remember I fixed a similar issue on hppa a few months ago. If we notice we are getting behind, we then schedule the next interrupt one tick later and count one more pending tick. This comparison was not made using signed arithmetic in the original code, so it was always true. The code thus thought it was getting late, and would schedule the next interrupt 1/hz second later. So instead of getting 100 interrupts per second and invoking hardclock on every interrupt, we were getting 50 interrupts per second and invoking hardclock twice on every interrupt. Timekeeping would be ok, but latency would be terrible. [...]
The next day, I made progress on the internal SCSI controller, but the results
were frustrating.
Date: Tue, 5 May 2009 22:13:14 +0000 From: Miod Vallat To: Joel Sing Subject: isp I've been concentrating my work on Fuel recently, and since a couple hours I have been able to get isp(4) to correctly see the disks and do I/O. So I was installing from the network, but eventually disk I/O froze while unpacking base45 (but the rest of the machine was alive, the ``stalled'' progress bar was being refreshed, machine would answer pings, etc). I need to investigate further (that's an ISP12160 chip on this machine, BTW). And I also need to clean up my changes, test them on Octane and try to go further.
A few fixes later, I was able to run multiuser on the Origin 200, but disk I/O
were extremely slow. The next step, besides figuring out the reason for that
terrible slowness, was to be able to boot from disk.
At first, I thought that, just like for the kernels due to different link
address on the various SGI families, there would be a need for one bootloader
binary per family.
But discussing this with other developers, Mark Kettenis pointed that
ARCLoad,
the bootloader now used by Linux, was documented to be able to load kernels on
IP27, IP28 and IP30 systems from a single binary.
I had a quick look at its code, and realized that the binary was built as a
relocatable file, instead of a standalone ELF binary. This way, it did not have
to require a specific address where to be loaded in memory, but would let
ARCS load it at the address of its choice.
Date: Sat, 9 May 2009 21:04:43 +0000 From: Miod Vallat To: Mark Kettenis Cc: Joel Sing, Theo de Raadt, Kenneth R. Westerback Subject: Re: sgi 64 bit bootblocks, second part > Hmm, http://www.linux-mips.org/wiki/ARCLoad claims the 64-bit version > supports IP27, IP28 and IP30, and looking at the source code, there > doesn't seem to be any model-specific switches in the makefiles beyond > the 32-bit vs. 64-bit switch. And it builds a relocatable file. So this means ARCBios is more smart [sic] than I thought (at least the 64 bit version). If this is true, and if I can get this to work, we'll need only two bootloaders (and the cdrom image will be bootable on all systems).
My next status update showed some progress.
Date: Sun, 10 May 2009 21:19:19 +0000
From: Miod Vallat
To: Joel Sing, Theo de Raadt, Jasper Lievisse Adriaanse
Subject: SGI status
Status update of the weekend.
IP30 / Octane:
still works fine. No network. There is a PCI cardcage on ebay
for USD 160, and the guy will only ship to the US, so it's not
worth looking for.
Known problems:
- root device computation is wrong - it is based on the
simplified O2 logic, while we need to move towards a
device_register() based logic. This is quite simple to do.
IP27 / Origin 200:
no improvement, on-board isp(4) is still slow. I swapped the
drive with one of the Octanes to confirm this is not a disk
problem. I believe some interrupts are stuck at the xbridge
level, but I can't confirm it yet; but I know what code to write
to confirm this. On the other hand that might be a problem
specific to isp(4), since the fxp(4) card I installed doesn't
have any problems.
Known problems:
- reboot or halt do not work - it looks like I need to flush all
tlb and restore ARCBios exception vectors before invoking
ARCBios to return to the prom.
- root device computation is wrong - same as Octane above.
IP35 / Origin 300: (tested by Jasper)
My heuristic to figure out the serial interrupt did not work on
this machine. I need to write brute force code (all pci
interrupts mapped to serial) and have Jasper test so that I know
the correct interrupt pin, then try and figure out why it is
different than on Fuel.
IP35 / Fuel:
I have been able to complete a make build, with network on
fxp(4), and storage on a 4GB SDHC card connected as umass(4),
with /usr/src on NFS and /usr/obj on umass. Build took 26 hours.
Unfortunately, when I wanted to install an ahc(4) pci card to
drive the local disk, I realized this machine has 3.3V PCI slots
and all my ahc cards are 5V.
The on-board isp(4) doesn't show the same lost interrupts issue
as on Origin 200, but after a while I/O return weird results
when reading back from the disk (causing ffs panics or sgivol
pretending the volume header does not exist), and the machine
freezes completely.
Known problems:
- same reboot problem as on Origin 200
- same root device problem as on Origin 200
- onboard isp(4) not reliable.
Boot blocks:
I have not been able to get relocatable binaries loaded by the
PROM. An arcload binary extracted from debian packages works,
the same arcload binary recompiled under OpenBSD fails with
relocation errors, likely due to toolchain differences. So I'll
settle with three non-relocatable bootblocks for now, and I'll
give this another try after I complete the binutils update.
Note that I am inclined to believe the isp(4) problems are
driver-specific, rather than pci controller-specific, as all the other
pci devices appear to work fine (fxp, ohci).
In the IP35 part of the message above, I mention a heuristic used to
figure out what is the proper interrupt for serial ports.
This is because the IOC3 chip, despite being attached to a PCI bus,
violates the PCI specification in multiple ways.
Depending on where the IOC3
chip was supposed to be used, it could be built with only a subset of its
components: for example, when put on the PCI CADDuo card, it would only contain
the PS/2 keyboard and mouse ports, and the Ethernet interface; but when put on
the MENET multi-Ethernet card, there would be four IOC3 chips, the first three
having only the Ethernet interface and the two serial ports, and the last one
having only the Ethernet interface.
Unfortunately, the SGI engineers who designed that chip decided to use two
interrupt lines, one for the Ethernet part, and one for all the other functions.
The PCI specification requires that a given PCI device can only use a single
interrupt line, unless it advertizes itself as a multi-function PCI
device (with up to eight functions), and each function can use a separate
interrupt line.
Of course, the IOC3 chip does not report itself as a multi-function
device, and there is no way to know that it uses a second interrupt, and
which one. And, just to make our lives difficult, that unknown second
interrupt, would be the non-Ethernet interrupt, the one needed to make the
serial console work.
If the Ethernet part was missing, of course, there would be a single interrupt
line and no trouble.
In these early days of IOC3 support, we were focusing on the on-board IOC3
found on all these machines, and since the interrupt assignment for the
on-board devices was fixed, I had the impression that it would use the next
unused interrupt pin for the second interrupt, because it appeared to work
that way on Octane and Origin 200. But this did not work on Fuel and I had
to use horrible tricks to detect the proper interrupt line to use.
This was the first and only time I used the word defecate in a
commit
message.
From: Miod Vallat
Date: Wed, 27 May 2009 19:04:47 +0000
To: openbsd-cvs
Subject: CVS: cvs.openbsd.org: src
CVSROOT: /cvs
Module name: src
Changes by: miod@cvs.openbsd.org 2009/05/27 13:04:47
Modified files:
sys/arch/sgi/pci: ioc.c
sys/arch/sgi/xbow: xbridge.c
Log message:
Yet another attempt at a more reliable detection of the second interrupt
used by onboard IOC chips, by forcing the IOC to trigger this interrupt,
and some help from the PCI bridge driver to report which interrupt has
fired through a fake PCI configuration register.
This works nicely on IP27 and IP35, but on IP30 the interrupt doesn't
happen, for some reason; so keep the existing heuristic in case the above
trick did not give us a valid interrupt number.
In case we got an interrupt, this will also detect IOC configurations where
there is actually one interrupt, should such configurations exist.
I probably deserve to rot in hell for this abomination, but I won't mind
as long as the IOC designers who came with the bright ``let's use more than
one interrupt and defecate on the pci spec'' ideas are there, too.
Eventually,
after a lot of testing, hair pulling, reading the Linux source code and
getting a CADDuo card on eBay, a
much
better algorithm was devised.
From: Miod Vallat
Date: Wed, 11 Nov 2009 15:29:31 +0000
To: openbsd-cvs
Subject: CVS: cvs.openbsd.org: src
CVSROOT: /cvs
Module name: src
Changes by: miod@cvs.openbsd.org 2009/11/11 08:29:31
Modified files:
sys/arch/sgi/pci: ioc.c
sys/arch/sgi/xbow: xbridge.c
Log message:
It turns out PCI IOC3 card which embed both the Ethernet controller and the
superio chip interrupt on two different pins (yet do not advertize
themselves as a multi-function device, of course).
So, on one hand, this makes the ioc attachment code simpler, because it
simply needs to map interrupt pins A and B, and another hand, this moves
all the interrupt knowledge to the PCI bridge driver, since routing of pin
B differs whether the device is the onboard IOC3 chip (and able to use
any of the 8 bridge interrupt sources...) or on a PCI board (with pin
mapping sane, since controlled by the bridge).
This makes superio interrupts on CADduo boards work. Tested to cause
no regressions on Origin 200, Octane and Fuel.
Despite what I wrote regarding the boot blocks, I suceeded in building
relocatable boot blocks for 64-bit ARCS, and they were added to the source tree
on may 14th.
The slow disk I/O performance on Origin 200 turned out to be a known SGI chip
errata, documented in IRIX headers, for which I had added the advised
workaround, but I had made a mistake in my code and it did not actually work
around the issue correctly. On may 15th, I fixed my code and suddently the
Origin 200 became usable.
Usable, but not stable.
Date: Thu, 21 May 2009 19:47:39 +0000 From: Miod Vallat To: Joel Sing Subject: sgi status [not including deraadt and kettenis as they are AFK] With the interrupt fixes in, things are in better shape, but there's still work to do. * Origin 200 The machine freezes for no apparent reason when compiling stuff or whatnot. I am able to reproduce this with `openssl speed -elapsed' systematically, it always happens at the same time. Even in single user. I need to write support for the NMI button, and then maybe I'll be able to figure out what's wrong. * Octane Still lots of problems with isp. Smells like a magic flag I forgot to set or clear in the bridge registers. This will hopefully work again when I'm done rewriting PCI resource allocation in xbridge. * Fuel All the problems I blamed on isp where actually caused by fxp. It turns out that, right now, I am trying to use the 2GB direct DMA window, and limiting bus_dmamem_alloc() to return memory in this range. However, mbuf are not allocated that way, and bus_dmamap_load_mbuf() - or even bus_dmamap_load() in the case of fxp - will be invoked on arbitrary memory. To fix this, I need to write support for the IOMMU part of the Bridge, and manage its ATE memory. However, this ATE memory is limited (especially on older Bridge vs XBridge). The Bridge can use ATE in external memory, but this memory is limited, and there is a bug which prevents that memory to be modified while there is a DMA transfer occuring, which means we can not really use it. So I'll write some iommu code to cope with situations like that, and upon lack of resources, the stack will try again later when resources have been freed. In the meantime I removed half the memory of the Fuel machine, so that all the physical memory is below the 2GB limit, and everything worked fine. The machine is currently baking a muild on local disk. * Left to do - xbridge pci resource management rewrite - xbridge iommu basics - nmi handling on ip27 - reboot code on ip27 and ip35 - integrate ip27 and ip30 kernels into release builds - more hw support (keyboard/mouse controller, video, ethernet...)
Shortly after this mail was sent, the build on Fuel completed.
yummy, that 600MHz SGI Fuel bakes a muild with src and obj on local disk in 6 hours and a half. which is about the same build time as a 400MHz hppa (B2000). nice! :) i think the 4MB L2 cache helps a lot.
(Note the expression “bakes a muild” above – the command used to
build a complete OpenBSD userland is “make build“, but since it makes
systems run hot and can take quite some time, at some point in a discussion
in october 2004 I jokingly exchanged letters to coin the “bake muild”
expression, and it has stuck to the point of becoming commonly used by other
developers since.)
On may 25th, after many experiments trying to figure out why the Origin 200
would eventually freeze under load, I had an epiphany and
found
the solution.
Years ago, I fixed an R5000 O2 instability by implementing a workaround for a chip bug, which was supposed to be fixed in that particular revision of the die but wasn't (tlbhandler.S 1.16). Being lazy, I did not write a runtime selection of the appropriate TLB handler code, although this was on my list. It turns out that this fix confuses the hell of R10000 processors revision 3 (but not earlier 2.x revisions), to the point of making the Origin 200 here hang so hard it would not even enter the NMI handler (don't ask me how I figured this was the cause). So it's time to choose the appropriate TLB handling flavour at runtime, building the trampoline code from the fixed exception handler location jumping to the handler address at runtime. As a bonus, kernels linked in KSEG0 get the address computation optimized and thus a smaller trampoline than before.
Interestingly, the R5000 errata commit I was referring to had also been done on
a may 25th, exactly two years before!
With the Origin 200 runnning stably now, I added the IP27 and IP30 kernel to the
OpenBSD/sgi builds on may 29th.
On june 2nd, support for these systems was
made
public.
Date: Tue, 02 Jun 2009 01:38:59 +0000 From: Miod Vallat To: openbsd-sgi Subject: Fuel, Octane and Origin 200 support Hello folks, I am glad to announce that we have reached a state where support for Fuel, Octane and Origin machines is worth testing. There are still known problems, which are being worked on, but these machines are however usable if you run the latest OpenBSD/sgi snapshot (binaries from May 31th). What works: - Fuel, Octane and Origin 200 boot multiuser. - On-board SCSI controllers are supported. - On-board USB controller on Fuel is supported. - PCI devices in Fuel and Origin 200 are expected to work. I use fxp(4) network cards in both Fuel and Origin 200, and will try more devices soonish. Some cards will not configure correctly because of limitations in my code, which I am working on. - For the Octane owners with PCI card cages, these should be supported as well as the Fuel and Origin 200 PCI slots, but this has not been tested. What doesn't work: - The onboard Ethernet interface on theses machines is not supported yet (which is why I am using fxp PCI network cards). This means that the Octane systems without PCI card cages are pretty useless at the moment. - Neither the PS/2 keyboard and mouse ports nor the graphics options are supported yet; serial console is required at the moment. - On multiprocessor machines, only one processor will be used at the moment. What might work: - Other systems of similar designs will probably work to some extent, but we won't know this until someone having access to these systems tests OpenBSD on them and reports how it went. Such systems are: Origin 2000, Onyx 2, Origin/Onyx 300, Origin/Onyx 3000. If your system has multiple nodes, only the resources of the first node will be used. What will not work: - The Origin/Onyx 350, Onyx 4 and Tezro could run, but there is no support for their PCI-X controller yet (which means no devices will be detected on them, not even the onboard devices). This will not change until a developer can get access to such a machine. Enjoy! Miod PS: I should thank Jean-Marc Harang for lending me a Fuel system to work on; this has helped immensely.
hmm, sgi feels much faster because of? because of speed! (-: probably the stuff that miod fixed a while ago interrupts? oh yes, that helped. getting faster machines to work also helped (-: am I halucianting? now you gotta get the even faster ones working. no kidding. where is the real miod is this reyk playing a joke on miod in return? holy fuck.. theo you better turn off miod's account. it's been compromised.
Theo de Raadt got a 700MHz R16000 Fuel system on june 16th, to use as a
build machine for the OpenBSD/sgi snapshots.
I also bought one (albeit 600MHz only)
and received it on june 23rd, which allowed me to return the
Fuel I had been using to its owner to complete the dare.
After setting up his machine with an
fxp Ethernet interface,
Theo de Raadt wanted to put a
bge
card in order to have a faster (gigabit) link. But the card
would not work, and he asked me whether I could fix this.
The reason why this board did not work is that this is a PCI64 card, able
to work with 64-bit wide bus addresses, and my code responsible for setting up
the PCI bus resources did not handle this particular case, for I had not thought
of testing that case. (To my defense, I am not sure I had any such PCI card to
test, to begin with.)
Reworking that code turned out to be rabbit’s hole, and I ended up eventually
writing complete code to set up the resources of a complete PCI bus and the
underlying buses, if any (PCI-PCI bridges, or PCI-Cardbus bridges.)
With the 4.6 release cutoff being close, I aimed for small changes, though,
and only a subset of these changes were commited on july 6th to make the
release, allowing bge cards to work. The remainder of the code, with support
for underlying buses, went in one week later, after the 4.6 release had been
tagged.
I thought it was time for a well-deserved break.
Date: Tue, 28 Jul 2009 18:35:44 +0000
From: Miod Vallat
To: Theo de Raadt, Jasper Lievisse Adriaanse, Joel Sing
Subject: non-O2 sgi status
I've been doing mostly sgi work in the last four months, and I'll need a
break from it shortly, to work on some MI stuff (wscons, etc) and spend
time on other architectures needing some love.
That's where you guys can step up (-:
Here is a short list of the problems I am aware of in the current
OpenBSD/sgi code for non-O2 machines:
* HARDWARE SUPPORT
The following two points need to be worked on as soon as possible:
- the serial ports do not work on Origin class machines, except for
the console port, while on Octane both serial ports work correctly.
I am currently investigating this, as this could be part of the
reasons why interrupt-driven serial output on Jasper's Origin 300
does not work.
- no onboard Ethernet controller support. This is a pre-MEC design, so
most of the MEC code can be reused, but since there is no per-txdesc
error information, we have to ask it to interrupt after each TX. No
wonder why its performance, even under IRIX, sucks.
The following points need to be worked on eventually, but it's not as
important as the above two at the moment.
- no keyboard/mouse controller support. This is almost trivial to do,
I'll probably give it a try unless Joel beats me to do it.
- video support for Impact-style graphics (found on Octane) and
VPro-style graphics (found on Octane and Fuel). There is code for
both in the linux-mips tree (or maybe with skylark's octane
patches), so it can be done eventually.
- PIC and IOC4 support for Origin 350 and Tezro support. Probably not
worth doing until the Origin 300 problems are fixed, because Origin
350 is likely to have the same. I think I have enough PIC
information, as for IOC4 I think I can support everything on it but
the atapi controller, which will need serious tinkering (but then
only the DVD drive is connected to it, so it's not vital for system
use).
* MI CONCERNS
We are only using memory under 2GB physical so far, because of DMA
concerns. The reason behind this is that the PCI bridge chip (driven
by xbridge(4) ) has a very limited number of IOMMU entries, so we are
falling back on a direct DMA window, which only spans 2GB.
This means that on multiple node Origin 200/2000 systems, we are only
using memory from the first node; and that on Fuel and Origin 300, we
are only using memory from the first bank.
However, devices able to use 64 bit DMA address can do so, and do not
need to go through the direct DMA window.
Ideally, we should be able to solve this in the MI code, by having the
MI code being able to know if the given mbuf or buf is going to go
through a DMA-limited device; and if so, check the DMA constraints and
perform bounce buffer allocation if needed. That's easier said than
done.
In the meantime, I can provide diffs which force mbuf pool allocators
to only use the low memory (but then devices such as the onboard
interface do 64 bit DMA so wouldn't need it, when supported), and to
only allocate physical memory for the buffers from the low memory (but
then, on machines with a lot of memory, we want the buffer cache to be
able to use more than 2GB of memory).
* SMP
The current code assumes it runs on CPU0. I plan to slowly move to
per-cpu_info interrupt information (i.e. interrupt mask and
acknowledge pointers, etc), so that if CPU0 is disabled or missing, we
can run on another processor.
Then I'll tackle the SMP work; however I'll probably want to take some
time to make the interrupt mess^Wcode cleaner (since that impacts the
to-be-written MP-compliant mutex code).
From my point of view, the most important thing we need during this
release cycle is hardware support, however. Being able to support the
on-board Ethernet will bring us many Octane users who can't find a
stable linux distro and are becoming fed up with IRIX, and even more so
if we have some decent glass console capabilities. Most Octane users
won't even notice the 2GB limit because there aren't many Octane systems
with more than 2GB; and they'll cope with the lack of SMP for a while.
Then of course to get fast build machines, it would be nice to make some
progress on Origin 300 and 350. But that's slightly less important.
Miod
PS: Oh, and I still intend to support the lower class IP22 machines
(Indy, Indigo 2) as long as they have processors without too many
hardware bugs in 64 bit mode. But I only have the boot blocks for them
so far (-:
(skylark in the message above being Stanislaw Skowronek.)
On september 1st, while searching for some Octane information, I stumbled upon
the blog of
Takuya Asada.
He had a dual-processor Octane 2 system, had noticed the OpenBSD work, and was
experimenting with multiprocessor support.
You can e.g. look at his posts during the
month of july
where most of them are focusing on the Octane.
This was too good to be true, so I contacted him immediately to ask whether he
would be interested to work on multiprocessor support with us.
Date: Tue, 1 Sep 2009 21:10:59 +0000 From: Miod Vallat To: Takuya Asada Cc: Jasper Lievisse Adriaanse, Joel Sing, Theo de Raadt Subject: SMP on OpenBSD/sgi Hello, I have just discovered your SMP work for OpenBSD on the Octane (on http://d.hatena.ne.jp/syuu1228/ ). Unfortunately I don't speak Japanese, and the output of the automatic translation tools is sometimes rough to read... If you don't recognize my name, I am one of the OpenBSD developers, and the person who wrote most of the Octane and Origin family support over the last few months. This is why your work has a lot of interest to me (-: I know from experience that adding SMP support to an OpenBSD architecture is a lot of tedious work, and I am really impressed by the work you have done already. Adding SMP to the sgi codebase, for both the Origin and the Octane systems, is on my list of things to do; and I was about to start doing preliminary work for this (writing SMP mutex code, adding locks around interrupts and traps, adding more information to struct cpu_info, etc). Most of this would duplicate work you have already done. In fact, some of the changes you have made are already found in the current OpenBSD source (such as astpending changed from a global variable to a per-process variable). Would you be interested in working more closely with us, and get your work integrated into the OpenBSD source tree? Regards, Miod PS: One important change I have in mind is to put the address of the per-cpu interrupt registers (on Octane, HEART_IMR(cpu) and HEART_ISR) in struct cpu_info, so that the interrupt code does not need to recompute the address every time. It would make sense to put the address of the IPI register there too. This will make the interrupt handling code faster.
Asada agreed with that offer, and was able to start working directly in the
OpenBSD source tree a few days later.
An OpenBSD mini
hackathon
was planned to happen during one week of november in
Coimbra, Portugal, and it was decided that I would ship the dual-processor
Octane I was using to the hackathon, and Takuya Asada, Joel Sing and I would
attend the hackathon to make progress on Octane multiprocessor support.
(I would end up working on Loongson hardware support instead during this
hackathon, and did not contribute much to their Octane work there.)
In the meantime, a Fuel was not good enough for Theo. And after that mail, he
was aiming for a multiprocessor Origin 350 system. Thanks to developer Ryan
McBride and Mark Uemura, a couple of such systems were found and
bought in Japan, and Theo was supposed to fly back to his lab with them in
september.
Unsurprisingly, the OpenBSD kernel did not run out of the box on the Origin 350.
A first difference between the Fuel and the Origin 350, both being of the IP35
family, was that the Origin 350 used a newer version of the horrible IOC3 chip.
That IOC4 chip was much better but needed a different driver; I had blindly
added a
tentative
driver
for that chip in august,
but it had never been tested until Theo tried it. After some painful debugging,
it turned out my code did not compute the serial port clocks correctly and was
programming wrong time divider values, preventing console input and output from
working.
Another difference between the Fuel and the Origin 350, was that the Origin 350
sported a more modern version of the XBridge chip, called the PIC.
Unlike the XBridge which supports a single PCI bus, the PIC supports two PCI-X
busses. The IRIX header files, as well as the IRIX code contributed to Linux
during the short time Linux had support for the Altix 350 (the Itanium-based
equivalent of the Origin 350), gave me enough information to blindly write
support
for the PIC
at the same time I had written the IOC4 code.
Once the kernel reached the point where we could get its output, on october 7th,
the PIC driver would attach, and its PCI subdevices were detected correctly.
But the machine would not boot much further – interrupts were apparently not
handled at all, and after a complete FIFO of messages (16 characters) was
output to the serial console, nothing happened.
Interrupt handling on these SGI systems is implemented in hardware through
message-passing, which was being transparent to the operating system; all it
needed to do was to write a proper destination address for interrupt messages in
a specific XBridge register, which allowed interrupts to either get lost (if
you didn’t program the register correctly), or sent to a particular Hub chip
associated to a particular CPU (or CPU pair, in the IP35 case).
The Brige, then XBridge, then PIC chip, was supposed to have 64-bit registers.
But a mistake in Bridge design caused 64-bit accesses to its registers to not
work; all accesses had to be done in 32-bit operations; and as actually
all registers only had no more than 32 useful bits, it was fine to perform
32-bit writes.
The PIC attempted to fix this. But while the fix restored proper 64-bit paths,
it actually broke 32-bit operation. So all PIC register operations had to be
done as 64-bit width load or stores, and this was easy to wrap with “read
register” and “write register” function pointers in the OpenBSD driver.
And then I had an illumination. During interrupt handling setup, a 64-bit
interrupt message destination address had to be written to a Bridge register.
But since the Bridge registers were actually only 32-bit, there were in fact two
registers involved to program this address, one for the “lower half” part of
the 64-bit value, and one for the “higher half”.
But the PIC, having working 64-bit registers, worked differently, by having the
“lower half” register, now of the proper width, accept the complete
64-bit value, and the “upper half” register ignore writes.
Date: Wed, 7 Oct 2009 15:29:07 +0000 From: Miod Vallat To: Theo de Raadt Subject: try this on O350 Magic interrupt register value was truncated to 32 bits, hence no interrupts from xbridge to cpu. That diff should fix it nicely. Oh, and I think I have found where the RTC is; I'll need to tinker in ddb later tonight. But right now I'm going to take a nap. [...]
My intuition of simply changing the code to write the complete 64-bit address to
the “lower half” register, regardless of the chip, made the Origin 350 unstuck
on october 10th, and a few hours later, Theo had already completed building an
OpenBSD/sgi snapshot and would use that machine as its build machine from then
on.
I could brag about announce Origin 350 support.
Date: Sun, 11 Oct 2009 20:22:38 +0000
From: Miod Vallat
To: sgi@openbsd.org
Subject: Origin 350 support
Hello folks,
I am glad to announce that, after a week of painful remote debugging,
OpenBSD/sgi now supports Origin 350 systems. In fact, the OpenBSD/sgi
snapshots are now built on an Origin 350, since a couple of days.
What prevented Origin 350 systems from working earlier was the lack of
support for their PCI-X bridges, as well as their base I/O board (IO8 or
IO9 boards). Both of these are now supported, although the IO8/IO9 ATAPI
interface, to which the DVD-ROM drive is connected, is not supported
yet.
This means similar systems, such as Onyx 350, Onyx 4 and Tezro should
work as well. if you are the lucky owner of such a system, please give
the latest OpenBSD/sgi snapshot a try, and let us know how it fares.
Enjoy!
Miod
PS: here is an Origin 350 dmesg for your enjoyment... note the processor
has 16MB of cache! That's gorgeous!
[ using 440792 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2009 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 4.6-current (GENERIC-IP27) #112: Sun Oct 11 12:46:15 MDT 2009
deraadt@sgi.openbsd.org:/sys/arch/sgi/compile/GENERIC-IP27
real mem = 2147483648 (2048MB)
rsvd mem = 47194112 (45MB)
avail mem = 1999663104 (1907MB)
mainbus0 at root
cpu0 at mainbus0 nasid 0: R16000 CPU rev 3.0 1000 MHz, R16000 FPU rev 10.11
cpu0: cache L1-I 32KB D 32KB 2 way, L2 16384KB 2 way
clock0 at mainbus0 nasid 0: ticker on int5 using count register
xbow0 at mainbus0 nasid 0: Bedrock revision 3
xbridge0 at xbow0 widget 15: PIC revision 3
pci0 at xbridge0 bus 0
iof0 at pci0 dev 0 function 0 "SGI IOC4" rev 0x53: irq 0
com0 at iof0 base 0x380: ns16550a, 16 byte fifo
com0: console
com1 at iof0 base 0x388: ns16550a, 16 byte fifo
com2 at iof0 base 0x390: ns16550a, 16 byte fifo
com3 at iof0 base 0x398: ns16550a, 16 byte fifo
iockbc at iof0 base 0x200 not configured
dsrtc0 at iof0 base 0x80000: DS1742W
isp0 at pci0 dev 2 function 0 "QLogic ISP12160" rev 0x06: irq 2
isp0: Board Type 12160, Chip Revision 0x6, loaded F/W Revision 10.4.41
scsibus0 at isp0: 16 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: SCSI3 0/direct fixed
sd0: 140014MB, 512 bytes/sec, 286749488 sec total
sd1 at scsibus0 targ 2 lun 0: SCSI3 0/direct fixed
sd1: 140014MB, 512 bytes/sec, 286749488 sec total
scsibus1 at isp0: 16 targets, initiator 0
bge0 at pci0 dev 3 function 0 "Broadcom BCM5701" rev 0x15, BCM5701 B5 (0x105): irq 3, address 08:00:69:11:e6:30
brgphy0 at bge0 phy 1: BCM5701 10/100/1000baseT PHY, rev. 0
pci1 at xbridge0 bus 0
bge1 at pci1 dev 0 function 0 "Broadcom BCM5704C" rev 0x03, BCM5704 A3 (0x2003): irq 0, address 00:e0:ed:07:b8:44
brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge2 at pci1 dev 0 function 1 "Broadcom BCM5704C" rev 0x03, BCM5704 A3 (0x2003): irq 4, address 00:e0:ed:07:b8:45
brgphy2 at bge2 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge3 at pci1 dev 1 function 0 "Broadcom BCM5704C" rev 0x03, BCM5704 A3 (0x2003): irq 1, address 00:e0:ed:07:b8:39
brgphy3 at bge3 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge4 at pci1 dev 1 function 1 "Broadcom BCM5704C" rev 0x03, BCM5704 A3 (0x2003): irq 5, address 00:e0:ed:07:b8:3a
brgphy4 at bge4 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
boot device: sd0
root on sd0a swap on sd0b dump on sd0b
At this point,
another OpenBSD developer who only had an Indy system, asked me whether these
systems could be supported. I told him this was on my list, but that I would not
work on it in the near future. He then asked for advice, in case he would try
his luck. I think my answer was interesting enough to be worth sharing here.
Date: Fri, 23 Oct 2009 04:52:34 +0000
From: Miod Vallat
To: other developer
Subject: Indy
Ok, here is a (not short enough) list of what you should know about the Indy.
* IP22/IP24
The Indy is actually IP24, but uses a similar logic board to the Indigo 2
(IP22), and actually reports itself via ARCBios as IP22. You'll eventually need
to tell IP22 and IP24 apart, since (for example) IP24 has no EISA slots.
Also there were flavors of these machines without frame buffers, intended to
be used as servers: the counterpart of the Indy is the Challenge S, while
the counterpart of the Indigo 2 is the Challenge M (Challenge L and XL are
different beasts).
There is code in NetBSD to figure out which particular flavour we are running
on. From now on, I'll use `IP22' to mean `any flavour' unless a particular
flavour is mentioned.
* Firmware
IP22 systems use a 32 bit ARCBios, which can only load ECOFF binaries. Since
our kernel is ELF, we need to provide ECOFF bootblocks.
Our bootblocks currently only support booting from disks. I plan on adding
the ability to load a kernel from the network, so that you could netboot the
bootblocks which would in turn netboot an ELF kernel.
However the ELF->ECOFF conversion of the bootblock image triggers an internal
bug in binutils; so far I am using a GPL ELF->ECOFF tool from the Linux arcload
(their sgi bootloader) sources, and I really ought to get that binutils
upgrade done ASAP so that hopefully I won't need that tool.
* Memory layout
On IP22, physical memory does not start at address zero. There's a 128MB gap
at the bottom of memory. From memory that's an area where the frame buffer
memory resides (but doesn't use the whole 128MB of course).
So this means the kernel needs to be linked at an address slightly after
128MB physical. You'll need to provide a new kernel, GENERIC-IP22, with
something like:
makeoption LINK_ADDRESS="0xffffffff88100000"
to cause it to be linked in KSEG0 after 128MB.
* Hardware components
NetBSD has support for all the Indy devices, except maybe for the onboard ISDN
controller (which we probably won't want to support anyway) and I think one of
the three Indy frame buffer options is not supported. Oh, and the Indycam
too.
IP24 has two GIO slots which allow for some expansion cards to be used (GIO
slots are also found on IP20, but not on IP22). Support for this can be
ported later.
The important devices are:
- the onboard device glue and interrupt logic (imc + hpc + ioc)
-- note that I didn't remember there was an ioc device on IP22 when I
wrote the IOC3 driver (a completely different chip) as ioc(4). For
IOC4 I named it as `IO Four' thus iof(4), so IOC should probably
become `IO One' thus ioo(4).
- the serial ports are driven by the usual Zilog 8530 chip. That's zs(4) on
many of our platforms, unfortunately they did not all track NetBSD up the
same point, so we have a bunch of different, but close to each other,
zs drivers on an MD basis, as well as a much older and buggier MI zs driver
in sys/dev/ic. I intend to clean this mess and only have one updated MI zs
driver, but have never found time to do it properly. Now would be the time...
- the keyboard and mouse ports are also handled by a Z8530. The devices
themselves are regular PS/2 devices.
- on-board Ethernet is based on a Seeq chip. I think a variant of this
chip was used on mac68k, but without DMA. It is best to port the NetBSD
driver, but (of course!) the driver won't port directly, as we use a
different structure to store the Ethernet address and generally do arp
related operations.
- on-board SCSI is the good old WD33C93 chip. This chip was popular at some
point in the workstation market, but did not get that widespread. We have
a very old driver for it for mvme68k, but it is associated with a very
special DMA controller so can't really be reused. On the NetBSD side,
they now have made the driver MI since it is used on a few platforms,
but there have been some issues so only their sgimips port uses the MI
driver yet. Some particular IBM disks confuse the driver when negotiating
the transfer characteristic (speed and sync vs async operation). I'd
advise starting from the MI driver nevertheless, and `update' it from
the NetBSD `scsipi' framework to our `scsi' framework. To help do this,
you should look at the NetBSD cvs history and see what kind of changes
were made when the thorpej-scsipi branch was merged and undo them (look
at a MD wd33c driver, though, as the MI driver has always been written
for scsipi. sys/arch/amiga/dev/sbic.c might be a good source of
inspiration).
So that's a lot of ungrateful tasks to do before you can have a kernel that
works. You should probably start with the serial stuff (just put yet another
copy of the netbsd mi code as md code in the port and wait for the janitor
to clean this...) and the onboard logic stuff. That would get you a kernel
able to print a few things on the console without needing the firmware for
this, and then dying because of the lack of root devices; and then you can
add ethernet and scsi to make this a bit more useful.
I'll clean up my bootblocks diff and send them to you this evening (actually,
I'm probably going to commit a few of them to make things even simpler).
(All this work, including driver cleanup, would eventually happen,
a few years later.)
After working on more changes to the interrupt code in order to better support
the various interrupt controllers found on the high-end systems, I came back to
the onboard Ethernet driver.
Date: Fri, 30 Oct 2009 19:16:29 +0000 From: Miod Vallat To: Jasper Lievisse Adriaanse, Joel Sing Subject: octane onboard ethernet progress X-Status: A I have resumed working on $SUBJECT. I am making progress - I now have the TX path working correctly (at least for short packets). But I can't receive anything yet. I'm currently hammering the RX code, and hopefully I'll have something sort-of-working soon.
And the next day:
Date: Sat, 31 Oct 2009 00:51:49 +0000 From: Miod Vallat To: Joel Sing Cc: Jasper Lievisse Adriaanse Subject: Re: octane onboard ethernet progress [...] > Well, that is still much further than I managed to get - care to share a diff? Of course. Do not mind debug code and bugs in there. My test setup so far consists of booting GENERIC-IP30 single user over the network, and then I have a machine which is not the bootp/tftp server, which runs ``tcpdump -x ether host''. This machine also has a static arp entry for the Octane, which allows me to send ping for as long as I want without getting `host is down' errors due to the lack of arp answers. On the Octane, when I ifconfig iec0 with an address, I see on the tcpdump machine the arp request followed by an icmp6 sol request. On the tcpdump machine, pinging the Octane shows rx interrupts occuring on the Octane, but with rx descriptor status being invalid, so all packets received are eaten as errors and nothing serious happens. > I have a large chunk of code that should work, minus the fact that I could > never get interrupts. We might be able to recycle or merge some code... The trick to get RX interrupts, is to enable the RX timer interrupt, with a zero timer. This means that we will get interrupts on packet received + timer expiration. With a timer of zero we get the interrupt immediately upon receive. You will probably curse me for choosing different symbolic names for the registers and their bits than you did (and they are also different from my earlier attempts). Miod PS: there is nothing preventing that code to work on GENERIC-IP27 kernels. However if you try it on an IP35 machine (Fuel, Origin 3*), the driver will not know its ethernet address at the moment. I know what to do to get the address, but this will be horrible (I will need to send commands over the internal L1 serial port access to get EEPROM contents and parse them), so I won't do this until the driver is working. [...]
After a few days of work, the driver was in working state and put in the tree.
But the RX performance was horrible, and I could not figure out why.
My code also had a bug causing every one out of 64 packets to be lost, and took
some time to figure out and fix.
I gave more details about it on the OpenBSD developers chat years later, which
is a good thing, because I no longer remember that particular bug!
here's an odd one: anyone seen a problem between a BBB and yp? if I run yp on one of my BBB's, it loses network in a few days. no yp, no network loss. Nick: that reminds me an evil sq(4/sgi) issue which was preventing packets from a certain size (which yp is fond of but noone else apparently is) to not get xmit. you might want to try all ping -s values and see if all sizes are working. sounds like a fun one, I'll do that. around 60 byte ethernet frames might be a toucy spot cos it has to pad the packet on ethernets behalf or this could be the iec(4/sgi) bug: every 64th packet get lost. this one was tough to fix miod: how big was the ring? 256 IIRC larger than 64 in any case. what was it? i botched programming the interrupt enable, so when it would hit the low threshold it would fail to interrupt for the given descriptor. miod: nice
On november 3rd, I eventually realized I had to
align
the RX descriptors to 4KB boundaries to get the proper RX performance.
This made the Octane much more useful, and came at the right time, with the
november hackathon approaching.
A few days later, on november 8, I had working code to query the L1 interface on
IP35 systems to get the onboard Ethernet address, and I could use the onboard
interface of my Fuel.
A side effect of the work toward supporting the Loongson hardware was that it
required a larger MMU page size, and the changes to support this in the generic
MIPS code allowed me to experiment with a larger page size (16KB instead of 4KB)
on the sgi systems as well. Larger page sizes reduce the pressure on the
address translation cache entries
(TLB),
which there is a limited number of.
wow, make build on sgi with 16KB pages takes 15% less time.
This speedup was unexpected (I was expecting about 3%)
and convinced me to try and use 16KB pages on
OpenBSD/sgi too.
Date: Wed, 2 Dec 2009 06:11:33 +0000 From: Miod Vallat To: private OpenBSD mailinglist Subject: mips64: support for 16KB page size The following diff adds support for kernels with 16KB page size, but does not change the page size yet, so this diff should be harmless at the moment except for possible bugs in it (I'm using it with both 4K and 16K page kernels at the moment). Details: -is now required to provide the page size in bits (PAGE_SHIFT). For now it remains 12, but you can play with making it 14 on sgi. - pmap and pte related defines moved away from - pmap structures which used to be 4KB and allocated as pages remain 4KB but are now allocated from a pool, which uses the previous page allocation functions as its own pool allocator. This causes a bit of overhead on 4KB page size kernels, but avoids wasting memory on 16KB page size kernels. - with 16KB pages, UPAGES is only 1, so instead of reserving a TLB pair and aligning the U area on an even TLB boundary, no TLB are reserved and we use a directly translated address which won't fault, for the U area (see vm_machdep.c, context.S) - tlb handling code will now restore the TLB entry mask when necessary, which is needed to handle > 4KB page sizes. Also check for tlbp instruction result correctly and do not consider tlb entry #0 special in any way. - no virtual cache aliasing occur with 16KB page sizes on most machines, so check for this in pmap_prefer(). [...] At any time, you can check what page size your kernel uses with $ sysctl hw.pagesize Miod
While I was tinkering with page sizes, Takuya Asada was slowly making progress
on SMP.
SMP on SGI worked, a few seconds ;) until it hit the scheduler? ;) it collapsed into a black hole under the weight of its own awesomeness that machine still aren't dead.
We were still far away from IP27 and IP35 SMP, and Theo would remind me on a
regular basis.
my sgi feels faster but still only about half the speed it should boo hoo that line is getting old. I think if miod keeps halving the distance between current performance and the maximum, Zeno has some thoughts on when the maximum will be achieved bwahahahahaha you just made my evening :-)
fall 2009 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP6, IP10 | Personal Iris 4D/2x | complete distribution no X server |
||
| IP12 | Indigo (R3000) | complete distribution | ||
| IP20 | Indigo (R4000) | complete distribution | ||
| IP22 | Indigo2 | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only |
|
| IP24 | Indy | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only |
|
| IP27 | Origin 200, Origin 2000 | complete distribution | complete distribution no SMP |
|
| IP28 | POWER Indigo2 R10000 | same as IP22 | ||
| IP30 | Octane | not-yet-integrated kernel patches X server on Impact only |
complete distribution no graphics no SMP |
|
| IP32 | O2 | complete distribution no R10000 support |
complete distribution | complete distribution |
| IP35 | Fuel, Origin 300, Origin 350, Origin 3000, Onyx 350, Onyx 4, Tezro | complete distribution no graphics no SMP Origin 300 and 3000 not supported |
[go back to the previous part][skip
to the next part]
2010, OpenBSD
The multiprocessing work eventually reached a state where the Octane
multiprocessor kernel was running stably, and it was enabled in the builds on
january 19th.
Joel Sing contributed a console video driver for the Odyssey frame buffer
(VPro) found on the Fuel and some high-end Octane systems in early march;
I took care of the other Octane frame buffer, the Impact, immediately after.
Date: Sun, 7 Mar 2010 20:54:06 +0000 From: Miod Vallat To: Theo de Raadt, Joel Sing Subject: sgi (octane) impact frame buffer driver This is the other framebuffer option available on Octane. Had I known I could have get it done in two days, I would have written it much earlier... So do we want this for the release? (this would allow all Octane users to be able to use glass console regardless of their setup). This diff conveniently forgets to add a framebuffer type for it (thus no wsconsio.h change) and a manpage, to avoid changing the tree outside arch/sgi. Miod PS: this is heavily based upon the linux code, with the command fifo programming made easier to understand, and putchar() able to use a 12x22 font, while the linux code does not work for width != 8.
We were getting close to the OpenBSD 4.7 release, so we had to hurry. I moved
the frame buffer of one of the two Octanes in my lab into the other, to create a
multihead setup, and check that the OpenBSD kernel would make the same decision
as the PROM to decide which of the two heads was the console.
Guess what, it didn’t and was needing some more code.
Date: Fri, 12 Mar 2010 16:46:26 +0000 From: Miod Vallat To: Joel Sing, Mark Kettenis, Theo de Raadt Subject: important fixes for multihead on sgi After putting two heads on an Octane, the following fixes turned out to be necessary: - MAKEDEV: create device nodes for up to four heads. - ip30_find_video(): when multiple heads are present, the PROM picks the highest widget as the console, not the lowest. - impact_attach(): allocate non-console screens with M_ZERO, otherwise we will not allocate backing store correctly and panic at the first output. Miod [...]
These changes were small enough to be allowed to make the release.
After that, there was not much sgi-specific activity for a while.
On his Origin 350 machine, Theo de Raadt noticed that, sometimes,
all disk I/O would stall, and after a hard reset, the filesystems on the disk
would often be so horribly corrupted that he would have to reinstall.
I suspected a lost interrupt, but could never figure out the real cause of the
problem (and therefore could not deliver a fix). I myself encountered this
issues only twice in more than five years after getting my own Origin 350 system
at the end of 2014. My machine using slower 700MHz processors compared to Theo’s
1GHz processors, this could also be a race condition somewhere, which would
explain why it happened much more often to him. But none of the tentative
changes I made to try and understand the problem helped.
2012, OpenBSD
You might remember the email I sent to another developer in october 2009 about
the work needed to run on Indy and Indigo2 systems?
I eventually decided it was time to roll up my sleeves and do the work.
Date: Sun, 18 Mar 2012 18:49:51 +0000
From: Miod Vallat
To: other developer
Subject: Re: Indy
It's been a long time since we last discussed porting to the Indy and
related systems.
As a convenient excuse to keep myself busy and unable to work on other
topics, I have started porting the NetBSD code. This is still a long way
from being usable, but here is a teaser, with loads of debug messages:
Running power-on diagnostics...
Starting up the system...
To perform system maintenance instead, press
Setting $netaddr to 10.0.1.210 (from server )
OpenBSD/sgi-IP22 ARCBios boot
arg 0: bootp()/bootecoff
arg 1: OSLoadOptions=auto
arg 2: ConsoleIn=serial(0)
arg 3: ConsoleOut=serial(0)
arg 4: SystemPartition=bootp()
arg 5: OSLoader=bootecoff
arg 6: OSLoadPartition=bootp()
arg 7: OSLoadFilename=/bsd.IP22
Boot: bootp()/bsd.IP22
Setting $netaddr to 10.0.1.210 (from server )
Setting $netaddr to 10.0.1.210 (from server )
1239328+552872 [100Setting $netaddr to 10.0.1.210 (from server )
+95496+52587]=0x1d9f28
ARCS32 Firmware Version 1.10
MEM 0, 0x8002000 to 0x8740000
MEM 1, 0x8800000 to 0x14000000
Found SGI-IP22, setting up.
Initial setup done, switching console.
[ using 149088 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
uvm_km_kmem_grow: grown to 0xc00000000c000000
OpenBSD 5.1-current (GENERIC-IP22) #32: Sun Mar 18 18:25:48 GMT 2012
miod@saliouse.gentiane.org:/usr/src/sys/arch/sgi/compile/GENERIC-IP22
real mem = 201326592 (192MB)
rsvd mem = 794624 (0MB)
avail mem = 191455232 (182MB)
mainbus0 at root: Indy
cpu0 at mainbus0: MIPS R5000 CPU rev 1.0 150 MHz, R5000 based FPC rev 1.0
cpu0: cache L1-I 32KB D 32KB 2 way
cpu0: L1 set size 16384:16384
cpu0: L1 line size 32:32
cpu0: Alias mask 0x3000
cpu0: Config Register 1043e6f3
cpu0: Cache configuration 2
cpu0: Status Register 100048a0
clock0 at mainbus0: ticker on int5 using count register
int0 at mainbus0 addr 0x1fbd9880
imc0 at mainbus0: revision 3, board rev 0
memory cfg: 2f407f20 00000000
gio0 at imc0
framebuffer at gio0 addr 0x1f000000 not configured
hpc0 at gio0 addr 0x1fb80000: SGI HPC3 (onboard)
Serial EEPROM:
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff 0xffff
zs0 at hpc0 offset 0x00059830
zstty0 at zs0 channel 1: console
zstty1 at zs0 channel 0
pckbc at hpc0 offset 0x00059840 not configured
sq at hpc0 offset 0x00054000 not configured
wdsc at hpc0 offset 0x00044000 not configured
haltwo at hpc0 offset 0x00058000 not configured
pione at hpc0 offset 0x00059800 not configured
panel at hpc0 offset 0x00059850 not configured
boot device: 'bootp()' unrecognized.
root device:
One week later, I gave him a status update.
Date: Sun, 25 Mar 2012 19:23:11 +0000
From: Miod Vallat
To: other developer
Subject: Re: Indy
Status update: at the moment I am running multiuser diskless on all my
R4400 and R5000 based systems. R4000-based systems exhibit very odd
crashes and what looks like memory corruption, so I don't know how much
of this is caused by these processors not being reliable in 64-bit mode,
and how much of this is caused by bugs in my shiny new cache routines.
There are issues with the Ethernet driver, though, some packets fail to
transmit with a `should never occur according to the specs' TX DMA
error. This causes ypbind to fail, and I currently have no idea what
causes this (except perhaps for bugs introduced while porting the
driver from NetBSD).
What is left to port is the SCSI driver, the frame buffer drivers, and
the few extras (Indy power button and volume controls, keyboard and
mouse ports, and Indy/Indigo2 audio). My intent is to commit this as
soon as the SCSI driver is working, and do the frame buffer / input
devices work later.
What is left to support as well:
- R4600 systems (I don't have any, a friend of mine has one so this will
get tested eventually)
- L2 cache on R4600 and R5000 Indy. At the moment, the L2 cache on the
R5000 Indy is disabled; I have work-in-progress code to handle it, but
this can't really fit in until my revamp of the mips cache routines is
ready and stable; but again, this can be completed later.
Mandatory boot log:
System Maintenance Menu
1) Start System
2) Install System Software
3) Run Diagnostics
4) Recover System
5) Enter Command Monitor
Option? 1
Starting up the system...
Setting $netaddr to 10.0.1.210 (from server )
OpenBSD/sgi-IP22 ARCBios boot
arg 0: bootp()/bootecoff
arg 1: OSLoadOptions=auto
arg 2: ConsoleIn=serial(0)
arg 3: ConsoleOut=serial(0)
arg 4: SystemPartition=bootp()
arg 5: OSLoader=bootecoff
arg 6: OSLoadPartition=bootp()
arg 7: OSLoadFilename=/bsd.IP22
Boot: bootp()/bsd.IP22
Setting $netaddr to 10.0.1.210 (from server )
Setting $netaddr to 10.0.1.210 (from server )
2824608+643136 [100Setting $netaddr to 10.0.1.210 (from server )
+175584+103501]=0x392ff8
ARCS32 Firmware Version 1.10
Found SGI-IP22, setting up.
Initial setup done, switching console.
[ using 280088 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
uvm_km_kmem_grow: grown to 0xc000000008000000
OpenBSD 5.1-current (GENERIC-IP22) #162: Sun Mar 25 19:11:16 GMT 2012
miod@saliouse.gentiane.org:/usr/src/sys/arch/sgi/compile/GENERIC-IP22
real mem = 134217728 (128MB)
rsvd mem = 802816 (0MB)
avail mem = 128237568 (122MB)
mainbus0 at root: Indy
cpu0 at mainbus0: MIPS R5000 CPU rev 1.0 150 MHz, R5000 based FPC rev 1.0
cpu0: cache L1-I 32KB D 32KB 2 way
clock0 at mainbus0: ticker on int5 using count register
int0 at mainbus0 addr 0x1fbd9880
imc0 at mainbus0: revision 3
gio0 at imc0
hpc0 at gio0 addr 0x1fb80000: SGI HPC3 (onboard)
zs0 at hpc0 offset 0x00059830 irq 29
zstty0 at zs0 channel 1: console
zstty1 at zs0 channel 0
pckbc at hpc0 offset 0x00059840 irq 28 not configured
sq0 at hpc0 offset 0x00054000 irq 3: Seeq 80c03, address 08:00:69:09:62:15
wdsc at hpc0 offset 0x00044000 irq 1 not configured
haltwo at hpc0 offset 0x00058000 irq 12 not configured
pione at hpc0 offset 0x00059800 irq 5 not configured
panel at hpc0 offset 0x00059850 irq 9 not configured
dsclock0 at hpc0 offset 0x00060000
vscsi0 at root
scsibus0 at vscsi0: 256 targets
softraid0 at root
scsibus1 at softraid0: 256 targets
boot device: 'bootp()' unrecognized.
root device: sq0
nfs_boot: using interface sq0, with revarp & bootparams
nfs_boot: client_addr=10.0.1.210
nfs_boot: server_addr=10.0.1.1 hostname=auze
root on 10.0.1.1:/netboot/rance/root
swap on 10.0.1.1:/netboot/rance/swap
Automatic boot in progress: starting file system checks.
setting tty flags
pf enabled
ddb.console: 0 -> 1
vm.swapencrypt.enable: 1 -> 0
starting network
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons: portmapnfs server odyssee:/netboot/rance/root/usr: not responding
nfs server odyssee:/netboot/rance/root/usr: is alive again
ypbindsq0: transmit underflow
.
Two days later, I sent him basic instructions to give that work a try.
Date: Tue, 27 Mar 2012 16:56:21 +0000 From: Miod Vallat To: other developer Subject: OpenBSD/sgi on Indy quick crash course Diskless and serial console only at the moment, and the L2 cache will not be used. But you can impress your friends! - get a recentish OpenBSD/sgi snapshot (at least base and etc). - get bootecoff and bsd.IP22 from http://miod.online.fr/sgi/ bsd.IP22 is a moving target, look for more files in there. - setup your NFS server mkdir -p /exports/indy echo "/exports -maproot=root -alldirs -network=-mask= " \ >> /etc/exports pgrep portmap || portmap pgrep mountd || mountd pgrep nfsd || nfsd pkill -HUP mountd nfsd echo " root= :/exports/indy" \ >> /etc/bootparams pgrep rpc.bootparamd && pkill rpc.bootparamd rpc.bootparamd cd /exports/indy tar xpzf /path/to/base51.tgz tar xpzf /path/to/etc51.tgz cd dev; sh ./MAKEDEV all (and of course you might want to setup hostname.sq0, hosts, resolv.conf, myname, mygate, etc). And change the root password, which is empty - but you can do it from the Indy later. - setup your tftp server mkdir -p /tftpboot uncomment tftpd line in inetd.conf and HUP it, or use dlg's rewrite which I have not tinkered with yet cp bootecoff bsd.IP22 /tftpboot - on the indy serial console abort the boot, pick `5' at the menu choice setenv netaddr setenv console d # to set up boot blocks path setenv SystemPartition bootp() setenv OSLoader bootecoff # kernel path setenv OSLoadPartition bootp() setenv OSLoadFilename bsd.IP22 then enter `exit', and choose `1'. What will happen: - the Indy will try to fetch OSLoader from SystemPartition. - SystemPartition being bootp() means network boot. - network boot will issue a reverse ARP request unless `netaddr' is set (this saves time) - bootecoff will get fetched from tftp - bootecoff will try to fetch OSLoadFilename from OSLoadPartition, again using tftp - the kernel will load, then there will be a pause while it gets read again for symbols (backwards seek do not exist in tftp). This will garble the display with several ``Setting $netaddr to ...'' messages. - the kernel will ask for its root device. Enter `sq0'. If the Indy does not tftp load: try to sysctl net.inet.ip.portlast=32767 but R5k Indy should not need this at all. Only older PROMs need this. In any case, `tcpdump ether host ' on the server can help.
I also wanted to get other developers’ opinions on support for some of these
systems.
Date: Tue, 27 Mar 2012 21:19:38 +0000
From: Miod Vallat
To: private OpenBSD mailinglist
Subject: mips: do we want to run on R4000 systems?
I have been working recently on something I had promised for years:
extending our OpenBSD/sgi port to the Indigo, Indy and Indigo2 family.
These systems used to be quite popular and there are still people who
have one of these systems but nothing more recent (such as an O2 or an
Octane).
Supporting these systems makes sense, from a hobbyist point of view, and
also to keep people from installing NetBSD on these machines.
Now that's all for the nice side of this work.
The bad news: the R4000 and, to a lesser extent, the R4400 processor,
are plagued by processor bugs, when running in 64-bit mode. Some of them
can be worked around in the kernel, others can be worked around in the
compiler.
At this point, we have the following three choices:
1) don't bother trying to support these systems.
pros: already done.
cons: will make miod and [other developer] unhappy.
2) only support R4600- and R5000- based systems, which do not suffer
from these horrible bugs. This means limiting support to the
``modern'' Indy systems only, since none of the Indigo and Indigo2
systems exist with these processors.
pros: it's just a bunch of rm in my Indigo tree...
cons: will make miod unhappy. Especially since I would like to port
to the R8000 and R10000-based Indigo2 systems in the future (assuming
I can get any, that is)
3) support all 64-bit capable processors, including the R4000 and R4400.
This requires the kernel and userland to be built with the following
options: -mfix-r4000 -mfix-r4400. These options reorder some
troublesome instructions to make sure there is no risk of triggering
a processor errata.
It is *very* important to compile everything with these options.
Without them, a kernel dies with a bogus assertwaitok panic within
seconds, because a M_NOWAIT argument becomes trashed into M_WAIT.
Without them, openssl hangs trying to draw an RSA key. So will sshd.
I have built the system with these options, and to my surprise this
does not cause a noticeable code size growth, nor does it seem to
make things run slower (on an R16000 cpu at least.)
So, if we are considering doing, this, it would make sense to make
these options enabled by default when building on big-endian mips
systems, so that packages could be used on R4k-based sgi, without
affecting loongson systems.
pros: allows us to support all 64-bit capable systems in the Ind*
families, and will make miod very happy.
cons: small side-effect of changing the gcc options on all sgi
systems.
What would you guys prefer?
Hardware support status:
- processors: R4000, R4400 and R5000 supported. L2 cache on R5000 not
supported yet (not enabled, code is in the works). R4600 support is
missing, coming very soon as well (joint work with R5000SC support).
Tested on: R4000PC (no L2), R4000SC, R4400SC, R5000SC. Will get tested
on R4600PC soon.
- on-board devices: serial ports, SCSI and Ethernet are supported.
frame buffers and input devices are next, as well as Indy/Indigo2
audio. No plans to support the parallel port.
There are spurious (but reliably reproduceable) Ethernet TX errors on
some systems, which I am investigating.
- expansion buses: SCSI and E++ Ethernet GIO options should be supported
(I have an E++ and it seems to work). Fast Ethernet GIO option not
supported yet (it's a de(4) behind a PCI-GIO bridge, driver can't be
ported blindly from NetBSD without having the hardware). EISA bus on
Indigo 2 is not supported yet, but I have work in progress code and
hope to be able to support it soon (unless R4000 and R4400 support is
ditched).
Mandatory bootlog. This is my Indigo R4000SC system, with a drive from
the ``expandable'' SCSI disk pile, used for testing until the scsi
controller becomes stable (it is now). Note how many kernels it took to
reach this state...
Starting up the system...
OpenBSD/sgi-IP20 ARCBios boot
arg 0: scsi(0)disk(2)rdisk(0)partition(8)/boot
arg 1: OSLoadOptions=auto
arg 2: ConsoleIn=serial(0)
arg 3: ConsoleOut=serial(0)
arg 4: SystemPartition=scsi(0)disk(2)rdisk(0)partition(8)
arg 5: OSLoader=boot
arg 6: OSLoadPartition=scsi(0)disk(2)rdisk(0)partition(0)
arg 7: OSLoadFilename=bsd
Boot: scsi(0)disk(2)rdisk(0)partition(0)bsd
2900720+643480 [100+180816+106161]=0x3a7978
ARCS32 Firmware Version 1.10
Found SGI-IP20, setting up.
Initial setup done, switching console.
[ using 287984 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
uvm_km_kmem_grow: grown to 0xc000000004000000
OpenBSD 5.1-current (GENERIC-IP22) #201: Tue Mar 27 20:37:09 GMT 2012
miod@saliouse.gentiane.org:/usr/src/sys/arch/sgi/compile/GENERIC-IP22
real mem = 67108864 (64MB)
rsvd mem = 802816 (0MB)
avail mem = 61571072 (58MB)
mainbus0 at root: Indigo
cpu0 at mainbus0: MIPS R4000 CPU rev 2.2 100 MHz, R4010 FPC rev 0.0
cpu0: cache L1-I 8KB D 8KB direct, L2 1024KB direct
clock0 at mainbus0: ticker on int5 using count register
int0 at mainbus0 addr 0x1fb801c0
imc0 at mainbus0: revision 1
gio0 at imc0
hpc0 at gio0 addr 0x1fb80000: SGI HPC1 (onboard)
zs0 at hpc0 offset 0x00000d10 irq 5
zstty0 at zs0 channel 1: console
zstty1 at zs0 channel 0
zs1 at hpc0 offset 0x00000d00 irq 5
zstty2 at zs1 channel 1
zstty3 at zs1 channel 0
sq0 at hpc0 offset 0x00000100 irq 3: Seeq 80c03, address 08:00:69:06:b8:f2
wdsc0 at hpc0 offset 0x00000122 irq 2: WD33C93B (20.0 MHz clock, BURST DMA)
wdsc0: microcode revision 0x0d, Fast SCSI
scsibus0 at wdsc0: 8 targets, initiator 0
sd0 at scsibus0 targ 2 lun 0: SCSI1 0/direct fixed
sd0: 496MB, 512 bytes/sector, 1015812 sectors
dpclock0 at hpc0 offset 0x00000e00
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
boot device: sd0
root on sd0a (125e5cc33e7dbdca.a) swap on sd0b dump on sd0b
Automatic boot in progress: starting file system checks.
setting tty flags
pf enabled
ddb.console: 0 -> 1
starting network
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons: portmap ypbind.
savecore: no core dump
checking quotas: done.
clearing /tmp
starting pre-securelevel daemons:.
setting kernel security level: kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd sendmail inetd sndiod.
starting local daemons: cron.
Mon Mar 26 05:44:14 MDT 2012
OpenBSD/sgi (clamouse.gentiane.org) (console)
login:
The general answer was roughly “whatever, we don’t care”, so this code
hit
the tree
on march 28th.
Investigating the Ethernet issues led to
an
interesting fix
on april 8th.
Be more careful when reprogramming the sq(4) DMA and PIO timing parameters; the current logic can be traced back to DaveM's intership at SGI in 1996, and are adequate for the hardware he had access to. However, ``recent'' Indigo2 and Indy systems are fit with a faster (33MHz instead of 25MHz) GIO64 bus, which need different timing parameters, and guess what? The PROM knows the right values to set. Since programming these timing registers was apparently only necessary for the Challenge S second interface: 1) only reprogram those registers on an IP24 (Indy, Challenge S) system. 2) pick proper values depending upon the actual GIO64 bus speed. Item #1 fixes Ethernet operation on Indigo2 (at least my teal R4400SC). Item #2 fixes Ethernet operation on my R5000SC Indy. For the record, programming unoptimal value caused `TX DMA underrun' errors (documented as `can't happen' in the HPC3 documentation, oh the irony), which could be reproduced reliably with ypbind(8).
I also hit an interesting hardware behaviour where reading from an important
hardware register, a “bus arbitration register”, which value should remain
constant unless written to, was sometimes returning a completely different value
than its actual value, as if the read cycle had not been performed correctly,
and stale bus data was returned instead of the actual data.
The kernel initialization code would read that register, set or clear some
specific bits, and write it back. But if the value being read in the first
place was incorrect, despite the register contents being correct, the kernel
would write a completely bogus value, which could cause some busses in the
system to be disabled or become nonresponsive.
This was fixed in
this
commit
by making the register write-only in the kernel, after making sure we’ve read a
correct value.
Reading the IMC bus arbitration register is not reliable, at least on IP20, and can return completely bogus values; writing these values back to the register can have unexpected and hilarious side effects, such as disabling the frame buffer. Workaround this `feature' by reading the register in a loop until we read twice the same value, and the value looks legit; then cache this value in a global variable and handle the register from now on, as a write-only register.
Once things were stabilized, I posted a summary of this work to the OpenBSD/sgi
mailinglist.
Date: Tue, 24 Apr 2012 21:58:40 +0000 From: Miod Vallat To: sgi@openbsd.org Subject: Support for R4k Indigo, Indy and Indigo2 added Hello, I am happy to announce that the current snapshots of OpenBSD support the IP20, IP22 and IP24 SGI systems. In other words: - R4000 and R4400 Indigo (IP20) - R4000, R4400 and R4600 Indigo2 and Challenge M (IP22) - all Indy and Challenge S (IP24) I know that several (many?) of you had been asking in the past, and I had always intended to work on this (didn't I get an R4000 Indigo, 12 years ago, for this purpose?). Wait no more! This has eventually happened. This work was helped by the existing NetBSD work, which provided basic device support (serial, Ethernet and SCSI). Of course running these systems in 64-bit mode opened quite a large Pandora box-of-problems, but we have reached a point where the system is running stably and able to rebuild itself. Oh, did I mention that I stumbled onto a few hardware bugs, such as extremely important device registers not always returning their value when being read? Anyways, what this really means is that you can wipe the dust off your system, plug it back, boot the OpenBSD installer and get a working system in less than half an hour (unless there really is a lot of dust to clean first). There are still a few rough edges, though. What works: - onboard serial, keyboard/mouse, SCSI and Ethernet controllers. Tested and confirmed to work. - all frame buffers are supposed to work, at least in console mode. However, only Newport (XL/XGE, found on Indy and Indigo2) and Express/Ultra (XZ/Elan/Extreme, found on Indigo, Indy and Indigo2) have been tested. The Light/Entry/Starter graphics on Indigo, as well as Impact on Indigo2, ought to work out of the box, but could not be tested (donations welcome). There is no X11 server for any of these frame buffers yet. - GIO E++ boards work. GIO32 SCSI boards ought to work too, but could not be tested. - the Challenge S second Ethernet interface (on the IO+ mezzanine) ought to work. What doesn't work (yet): - the extra SCSI controllers (WD33C95) on the Challenge S IO+ mezzanine are not supported (anyone got a Challenge S to spare?) - Fast Ethernet GIO options (Phobos G130 and G160, as well as `Set Engineering' Fast Ethernet are not supported. There is code to borrow from NetBSD, but it's not really an option without access to the hardware. - on-board audio on Indigo (hdsp) and Indy/Indigo2 (haltwo). - on-board parallel port (honestly, I couldn't care less). - L2 cache on R4600SC and R5000SC processor modules. I am working on this, but need to introduce a few interfaces first, and this takes time checking I do not break other systems. Coming soon. What sort of works: - the EISA bus on the Indigo2 attaches and cards get detected. However I have only been able to test it with 3Com ep(4) boards; the 10MBit/s models have abysmal performance, and the 100MBit/s 3C597 (or Phobos G100) doesn't seem to interrupt. Your mileage may vary. Snapshots are available from OpenBSD FTP mirrors near you; do not hesistate and play with them! However, please, please, pretty please, do not expose an R4000 or R4400 based system to the internet. These processors suffer from unfixed errata which can be used by local users to gain supervisor privileges. On these systems, you can't trust your local users - at all. As in, never give me a shell access to the system or I'll root it without even thinking. R4600 and R5000 based systems do not suffer from such problems. If you have an early R4000 processor (either an 1.x or 2.x version), other bugs will prevent it from running stably (the so-called ``end of page'' bugs). The system will run multiuser, but from time to time, some processes will misbehave. My own Indigo has a revision 2.2 R4000, so I am experiencing these issues first hand (and this system is currently not able to rebuild itself because of these processor bugs). However the processor errata documentation is public, and I am working on designing a good way to circumvent them (and it's no easy task, especially with the goal of not affecting the performance of other processors). But enough said. The real reason for this work was to get good device support, in order to be able to port OpenBSD to the Power Indigo2 systems: both the R8000 flavour (IP26) and the R10000 flavour (IP28). Those systems come with a specific memory controller which needs some care in order to operate properly; and of course the R8000 processor is an odd beast which, to this day, is not supported by any free operating system. I would like to change this state of things, and have IP26 and IP28 systems be able to run OpenBSD, if only for the sake of it and the joy of getting my code to run on formerly unsupported hardware. Unfortunately, I do not own any such system - neither an R8000 Indigo2 nor an R10000 Indigo2. If you know of such a system collecting dust, which owner would not mind parting of, please let me know. I'm sure we can arrange something. Miod PS: If you only have R4k ``PC'' processors, that is, without a secondary cache... then do yourself a favour and don't try to compile anything on them... these small caches get blown a few orders of magnitude by gcc 4 when compiling even the smallest program. This is very hard on the R4000PC systems, which only have 2x8KB cache (I have such a system, it's perfectly stable but glacial as soon as you are compiling anything on it). R4400PC, R4600PC will be a bit more bearable because their cache is twice larger.
This led to an interesting discussion with Steve Rumble about how to
handle the R4000 end-of-page errata.
Technical details you may skip!
The R4000 end-of-page errata affects R4000 processors before the 3.x
revision. On these processors, if the last instruction of a MMU page is a
jump instruction, and the next page does not have a valid TLB entry,
in certain circumstances documented in the errata sheet, the processor will not
service the TLB miss exception correctly. This usually manifests as a nested
exception being taken and causing the userland program to terminate with
prejudice a segmentation fault.
Date: Thu, 26 Apr 2012 20:15:51 +0000 From: Miod Vallat To: Stephen M. Rumble Subject: Re: Support for R4k Indigo, Indy and Indigo2 added [...] R4000 EOP pages are easy to workaround, really. There are multiple ways to do it, but what worries me is the cost of them, speedwise. One way to do it is to check all pages for the troublesome instructions when they are pmap_enter'ed with PROT_X, and force the next page to always be mapped in a TLB if the troublesome page is mapped. Using two wired TLB pairs, this can be done. But this adds overhead because you can no longer insert random TLB entries in the TLB miss handler, you first need to check if these concern entries which are in the wired slots. There is also work to do if you have several EOP-erratae pages in sequence, because of the limited number of wired entries. So if you have two contiguous EOP pages being an odd page (second half of wired tlb #0) and an even page (first page of wired tlb #1), you need to also map the second page of wired tlb #1. But what if it's also an EOP-errata page? Then you need to map the page next to it, with yet another wired tlb entry... A way to block this chain is to map this page to a dedicated page full of break instructions. Then if execution goes from the first EOP page (last of tlb #0) to the second one (first of tlb #1), then to the third, it will hit a break instruction and the kernel can realize that the original EOP page which caused this wired TLB arrangment no longer needs to be mapped (and actually needs to be unmapped). But this will cause a lot of overhead in the TLB miss handlers. A different, easier to implement, way to do this, is to overwrite the potential errata triggering jump instruction with a special break instruction, and remember the original instruction in the pcb. This avoids the need for mapping the next page, but on the other hand this will cause extra physical memory usage due to copy-on-write. And you still have the problem of emulating the delay slot instruction because the MIPS exception handling does not allow delay slots to be recovered (unlike, say, sparc, where the exception code returns after setting the addresses of the next two instructions to execute: the delay slot and the branch destination). I am still thinking about this, maybe there are different ways to achieve this. Miod
Date: Fri, 27 Apr 2012 04:40:31 +0000 From: Miod Vallat To: "Stephen M. Rumble" Subject: Re: Support for R4k Indigo, Indy and Indigo2 added [...] > > One way to do it is to check all pages for the troublesome instructions > > when they are pmap_enter'ed with PROT_X, and force the next page to > > always be mapped in a TLB if the troublesome page is mapped. Using two > > wired TLB pairs, this can be done. But this adds overhead because you > > can no longer insert random TLB entries in the TLB miss handler, you > > first need to check if these concern entries which are in the wired > > slots. There is also work to do if you have several EOP-erratae pages in > > sequence, because of the limited number of wired entries. So if you > > have two contiguous EOP pages being an odd page (second half of wired > > tlb #0) and an even page (first page of wired tlb #1), you need to also > > map the second page of wired tlb #1. But what if it's also an EOP-errata > > page? Then you need to map the page next to it, with yet another wired > > tlb entry... > > I think that SGI took this approach with a limit on the number of EOP > pages in a process. There was some documentation I came across long ago > warning that a few really old binaries might not work properly because > of too many of these consecutive evil pages. That would make sense. I have added code to check for such pages and warn on the console, and got spammed by messages; however I don't remember seeing consecutive pages having the problem. Also, I was using a hammer and reporting all pages with a jump instruction as the last word of the page, regardless of the preceding instructions. According to the errata the errata will only trigger if there are pending loads before the jump, but this makes deciding the risk of errata tricky, and I don't want to go there yet. Turns out there are several pages in libc which end up with jumps, so all binaries are being reported as errata-prone. On the other hand, we are using 16KB pages which cuts the odds of triggering the errata by four. > > A way to block this chain is to map this page to a dedicated page full > > of break instructions. Then if execution goes from the first EOP page > > (last of tlb #0) to the second one (first of tlb #1), then to the third, > > it will hit a break instruction and the kernel can realize that the > > original EOP page which caused this wired TLB arrangment no longer needs > > to be mapped (and actually needs to be unmapped). But this will cause a > > lot of overhead in the TLB miss handlers. > > That's a clever solution. It'd be interesting to see how many such bad > pages actually show up in practice. It might be much worse today with > larger binaries and libraries than it was 20 years ago. I'd say the worst issue here would be jump tables in kernel code, or other rodata merged with text. There would be the risk of code in the EOP-errata page referencing memory in the page which temporarily points to the break instructions. The data access will not fault but return wrong data. Curse these unified TLBs! [...] > I think the simplest fix, especially as far as the kernel is concerned, > would be to have the toolchain generate jumps and branches only at > 8-byte aligned addresses. I believe that this is what SGI did, or > perhaps they were more precise in banning the end-of-page jump/branch. > I looked into hacking up gcc to emit .align directives around jumps a > long time ago, but was out of my depth and didn't make much progress. This is a no-go in my opinion, if only because, by default, the assembler will do an optimization pass itself and reorder instructions to get faster code. [...]
In the second half of may, I managed to source a POWER Indigo2 R10000 (IP28) to
try my luck with it.
An important difference between the R4000-based Indigo2 (IP22)
and the other models, be they R8000-based (IP26) or R10000-based (IP28),
is that the latter have an
ECC memory subsystem.
But the way the ECC memory controller works, it needs to perform extra
operations if there are memory accesses bypassing the processor’s cache,
i.e. if the processor performs uncached memory accesses.
In order to not permanently inflict the cost penalty of these extra operations,
the controller works in two modes:
- a “fast” mode, where, as the name suggests, it runs as fast as
possible, but uncached memory accesses are prohibited and will always fault. - a “slow” mode, in which uncached accesses to memory are allowed.
The machine boots in slow mode, but once the operating system kernel has
performed its initialization, it is expected to switch to fast mode to perform
better.
The Indigo2 onboard Ethernet driver accesses its Ethernet buffer
descriptors using uncached memory accesses. This would only work in slow
mode, so porting to the IP28 design required some work in order to make this
transparent to the driver code. I ended up modifying the driver to fetch
descriptor data, alter it if needed, and store (write) it back; when running
on the IP22 systems, the fetch and store operations would do nothing and the
returned address would be the address of the descriptor in uncached memory,
while on IP28 a copy of the descriptor in cached memory would be made, and
written back later. This allowed me to narrow the moments where I had to
switch back to slow mode to the smallest possible time frames.
In my archives, the oldest traces of a kernel booting on it appear in a
discussion about how the
Insite
Floptical
drives were being probed.
Date: Fri, 25 May 2012 08:27:35 +0000
From: Miod Vallat
To: Sebastian Reitenbach
Subject: Re: Support for R4k Indigo, Indy and Indigo2 added
> Do you have a floptical drive? I guess I should be able to remove it,
> without harm to the system, and could bring it with me for you to g2k12.
Does this match what you were seeing:
> bootp()boot64 bootp()bsd.IP28 --a
Setting $netaddr to 10.0.1.9 (from server )
Obtaining boot64 from server
1024+36624Setting $netaddr to 10.0.1.9 (from server )
+3824+1368+320Setting $netaddr to 10.0.1.9 (from server )
Setting $netaddr to 10.0.1.9 (from server )
entry: 0x900000002fff4e70
OpenBSD/sgi-IP28 ARCBios boot version 1.1
arg 0: bootp()boot64
arg 1: bootp()bsd.IP28
arg 2: --a
arg 3: ConsoleIn=serial(0)
arg 4: ConsoleOut=serial(0)
arg 5: SystemPartition=scsi(0)disk(1)rdisk(0)partition(8)
arg 6: OSLoader=sash
Boot: bootp()bsd.IP28
Setting $netaddr to 10.0.1.9 (from server )
Obtaining bsd.IP28 from server
Setting $netaddr to 10.0.1.9 (from server )
Obtaining bsd.IP28 from server
3262800+915560 [91Setting $netaddr to 10.0.1.9 (from server )
Obtaining bsd.IP28 from server
+193752+115291]=0x447c90
ARCS64 Firmware Version 64.0
Found SGI-IP28, setting up.
Initial setup done, switching console.
[ using 309976 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
uvm_km_kmem_grow: grown to 0xc000000010000000
OpenBSD 5.1-current (GENERIC-IP28) #0: Fri May 25 08:20:58 GMT 2012
miod@saliouse.gentiane.org:/usr/src/sys/arch/sgi/compile/GENERIC-IP28
real mem = 268435456 (256MB)
rsvd mem = 1064960 (2MB)
avail mem = 260440064 (248MB)
mainbus0 at root: POWER Indigo2 R10000
cpu0 at mainbus0: MIPS R10000 CPU rev 2.5 194 MHz, R10000 FPU rev 0.0
cpu0: cache L1-I 32KB D 32KB 2 way, L2 1024KB 2 way
clock0 at mainbus0: ticker on int5 using count register
int0 at mainbus0 addr 0x1fbd9000
imc0 at mainbus0: revision 5
gio0 at imc0
hpc0 at gio0 addr 0x1fb80000: SGI HPC3 (onboard)
zs0 at hpc0 offset 0x00059830 irq 29: 85230
zstty0 at zs0 channel 1: console
zstty1 at zs0 channel 0
pckbc0 at hpc0 offset 0x00059840 irq 28
sq0 at hpc0 offset 0x00054000 irq 3: Seeq 80c03, address 08:00:69:07:f0:e2
wdsc0 at hpc0 offset 0x00044000 irq 1: WD33C93B, 20.0 MHz, burst DMA
wdsc0: microcode revision 0x0d, fast SCSI
scsibus0 at wdsc0: 8 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: SCSI2 0/direct fixed serial.SGI_Seagate_ST11200N00637118
sd0: 1014MB, 512 bytes/sector, 2076777 sectors
sd1 at scsibus0 targ 2 lun 0: SCSI1 0/direct removable
probe(wdsc0:2:1): wdsc0: timed out; asr=0x00 [acb 0x9800000020dfc000 (flags 0x1, dleft 24)], probe(wdsc0:2:1): ABORT in timeout: csr=0x85, asr=0x00
probe(wdsc0:2:1): wdsc0: timed out; asr=0x00 [acb 0x9800000020dfc000 (flags 0x41, dleft 24)], probe(wdsc0:2:1): ABORT in timeout: csr=0x85, asr=0x00
The code was
commited
a bit later
(with the Floptical removed from my machine for the time being, until
some
bugfixes
in the SCSI controller driver solved the issue) on the same day.
Support for the POWER Indigo2 R10000 systems (IP28). Currently running with ECC checking disabled, which allows the existing Indigo2 drivers to run unmodified.
In order to support the many processor combination on the Indy and
Indigo2
systems, I needed to replace the cache handling routines, of which there was a
set per processor model, from almost unreadable assembly code to C code.
I sent a mail to explain why.
Date: Mon, 28 May 2012 19:25:32 +0000 From: Miod Vallat To: private OpenBSD mailinglist Subject: A note about mips cache routines Just a friendly reminder! In case you didn't notice, I have started, over the last few weeks, to replace assembly cache routines for MIPS processors, with C code. I have three good reasons to do this: - all this assembly code roots in 32 bit MIPS code written in 1993-1994 or so. - while gcc 2.7 would not compile C code to something as good as the assembly code we have, 15+ years later, this is definitely not the case. And if the C code is only slightly behind, it's way more readable. - rewriting this code as C is easier to maintain, and will allow me to, eventually, provide tailor-built routines on a `what processor are we running on today?'-basis. That third point by itself is the main reason behind this work, because R5000 routines have become a mess (3 hierarchies of cache, of which L2 and L3 may not exist, and L2 may not be implemented the same way, and thus need two different code paths). Once everything is in C code, it will be easy to have the ``MI'' (well, mips64-specific but port-agnostic) cache routines being curcpu() function pointers. This will finally allow me to write the necessary code for L2 cache on R4600 and R5000 SGI Indy systems, without disrupting other platforms too much (and then I might be able to convince jsg@ to buy me a beer or two, next time we meet). <-- that's my #1 reason to do this, of course Doing this is taking a bit longer than expected, because 1) it's a real PITA. 2) I can't help but want to fix or improve things in the process, yet this has to wait until the assembly code is converted to bug-identical C code; then we'll be able to improve things. This is known as the `art syndrome'. Example: the R10000 code we have assumes the L2 line size is 64 bytes. Well, this was true on O2 R10000 systems, but higher-end systems (Origin, Octane, Fuel...) use 128 bytes per line. This means we can actually make some L2 routines run faster on those systems... eventually. So while I have faster code ready, I first need to confirm the `iso' code works, before improving on it. In particular, there is growing evidence that we need some cache routines to operate on virtually indexed caches only (to avoid cache aliasing and satisfy pmap_prefer when applicable), while some really need to operate on all cache levels (when writebacks are really supposed to hit memory, and not an intermediate cache level). This mess will be definitely easier to clean once all cache code is converted to tame C code. I apologize in advance for possible broken kernels. If you run into trouble with the kernels within the next few days (make that half a month), please let me know. Oh, and of course, make sure to rm cache_*.d from your kernel build directories and rerun config, every time I commit new cache code. Miod
This led to the R4600SC and R5000SC having their external cache
supported.
Code for the external L2 cache controller on Indy/Indigo2 R4600SC and Indy R5000SC processor modules; these sport an up to 512KB, physically indexed, write-through L2 cache which is not connected to the canonical external cache interface of these processors (hence requiring specific code to drive it). The cache is enabled early and disabled before returning to ARCBios (for very nasty things happen otherwise). Tested on R5000SC, will be tested on R4600SC soon.
I could then move onto my next goal: running on the POWER Indigo2 R8000 (IP26).
These are extremely rare and only appear on the 2nd-hand market once in a blue
moon, so I bought one from
Ian Mapleson in the UK.
I initially aimed for example configuration #31 on his
Indigo2
list,
but he offered me #30 for the same price (i.e. extra 128MB of memory). With
shipping costs, I ended up paying UKP 340 for it, much less that the current
(2026) prices… (remember than Ian makes a living providing high quality
systems and spares, with a warranty, and SGI hardware in working condition is
becoming increasingly rare those days.)
I collected the machine from the local UPS dispatching center on july 3rd,
and started working on it a few days later.
The
R8000 processor is quite
different from the other MIPS processors. It has
some unique features, such as not being able to run in 32-bit mode at all,
the ability to use a different page size in the kernel than in userland,
and a completely different TLB organization than the other 64-bit MIPS
processors (in 3 banks of 128 independent entries).
It is also a highly superscalar architecture, able to schedule up to four
instructions in parallel (two integer instructions and two floating-point
instructions), but still performing in-order execution, which requires
significant work from the compiler to order instructions in order to be able
to issue as many instructions as possible in parallel.
With a good compiler, such as SGI’s
MipsPro, the floating-point
performance was impressive, and in fact, the code name for that processor was
TFP, for Tremendous Floating-Point.
For a very long time, there was no public documentation for this processor.
Thanks to the efforts of the linux-mips people, scans of a manual were
released with permission to distribute them, and this allowed me to work on
supporting that processor; various comments and macros in IRIX header files
documented the processor errata and workarounds required to run stably.
Another particularity of that processor is that some exception conditions which
are usually reported as internal exceptions on the other MIPS processors, are
reported as external interrupts. All these details required some plumbing and
internal reworkings of the generic MIPS code in the OpenBSD kernel, before I
could try and run a kernel on my new toy system.
Cache memory operation on the R8000 is also unusual, to say the least. First,
its internal (L1) cache is write-through only, and systems built around that
processor, of which there are only two: the POWER Indigo2 R8000
(IP26) and the multiprocessor POWER Challenge R8000 and POWER Onyx R8000 (IP21),
were using large and fast write-back L2 caches to compensate for this.
Second, there is no documented way to invalidate the instruction cache
(or a subset of it), and OpenBSD had no other choice than dedicate
a
page full of nop instructions
to fill the instruction cache and eventually evict its previous contents.
Date: Tue, 10 Jul 2012 20:58:02 +0000
From: Miod Vallat
To: Theo de Raadt
Subject: I think I am hitting a new level of madness
This is what I have been working in the last two days. Still far from
complete.
>> bootp()boot64 bootp()bsd.IP26
Setting $netaddr to 10.0.1.228 (from server )
Obtaining boot64 from server
1024+35376Setting $netaddr to 10.0.1.228 (from server )
+3920+1400+304Setting $netaddr to 10.0.1.228 (from server )
Setting $netaddr to 10.0.1.228 (from server )
entry: 0xa80000003fff52e0
bios_base = 0x9800000000001000
OpenBSD/sgi-IP26 ARCBios boot version 1.3
arg 0: bootp()boot64
arg 1: bootp()bsd.IP26
arg 2: ConsoleIn=serial(0)
arg 3: ConsoleOut=serial(0)
arg 4: SystemPartition=scsi(0)disk(1)rdisk(0)partition(8)
arg 5: OSLoader=sash
arg 6: OSLoadPartition=scsi(0)disk(1)rdisk(0)partition(0)
arg 7: OSLoadFilename=/unix
Boot: bootp()bsd.IP26
Setting $netaddr to 10.0.1.228 (from server )
Obtaining bsd.IP26 from server
Setting $netaddr to 10.0.1.228 (from server )
Obtaining bsd.IP26 from server
3310176+751488 [91Setting $netaddr to 10.0.1.228 (from server )
Obtaining bsd.IP26 from server
+192696+114541]=0x42ada8
ARCS64 Firmware
Found SGI-IP26, setting up.
ip22_ecc_init: not working on IP26 yet
Initial setup done, switching console.
[ using 308168 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 5.2-beta (GENERIC-IP26) #43: Tue Jul 10 20:40:31 GMT 2012
miod@saliouse.gentiane.org:/usr/src/tfp/sys/arch/sgi/compile/GENERIC-IP26
real mem = 671088640 (640MB)
rsvd mem = 1064960 (2MB)
avail mem = 660078592 (629MB)
mainbus0 at root: POWER Indigo2 R8000
cpu0 at mainbus0: MIPS R8000 CPU rev 0.0 75 MHz, R8010 FPU rev 0.1
cpu0: cache L1-I 16KB D 16KB direct, L2 2048KB direct
cpu0: L1 set size 16384:16384
cpu0: L1 line size 32:32
cpu0: L2 line size 128
cpu0: cache configuration 0
cpu0: virtual alias mask 0x0
cpu0: config register 00000000000088b0, status register 0000000014034820
clock0 at mainbus0: ticker on int5 using count register
int0 at mainbus0 addr 0x1fbd9000
imc0 at mainbus0: revision 5
gio0 at imc0
grtwo0 at gio0 addr 0x1f000000: GU1-Extreme
grtwo0: device has not been setup by firmware!
hpc0 at gio0 addr 0x1fb80000: SGI HPC3 (onboard)
zs0 at hpc0 offset 0x00059830 irq 29: 85230
zstty0 at zs0 channel 1: console
zstty1 at zs0 channel 0
pckbc0 at hpc0 offset 0x00059840 irq 28
sq0 at hpc0 offset 0x00054000 irq 3: Seeq 80c03, address 08:00:69:08:6a:35
wdsc0 at hpc0 offset 0x00044000 irq 1: WD33C93B, 20.0 MHz, burst DMA
wdsc0: microcode revision 0x0d, fast SCSI
scsibus0 at wdsc0: 8 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: SCSI3 0/direct fixed eui.000c50fffe3d119a
sd0: 34732MB, 512 bytes/sector, 71132000 sectors
wdsc1 at hpc0 offset 0x0004c000 irq 2: WD33C93B, 20.0 MHz, burst DMA
wdsc1: microcode revision 0x0d, fast SCSI
scsibus1 at wdsc1: 8 targets, initiator 0
haltwo at hpc0 offset 0x00058000 irq 12 not configured
pione at hpc0 offset 0x00059800 irq 5 not configured
panel0 at hpc0 offset 0x00059850 irq 9: power button
dsclock0 at hpc0 offset 0x00060000
eisa0 at imc0 irq 27
bus error: cpu_stat 00000000 addr 1fa000e4, gio_stat 00000000 addr 1fbd9007
Stopped at Debugger+0x4: jr ra
[...]
ddb>
Later that month:
mmh always nice when the sgi powers off.. much less noise. balony! it all music to miod's ears ah, you can't understand. this machine he is complaining about used to be mine. some bits of our sgi port where written on this very machine. hence its continued screams of pain? it's a nice box... but noiser than an Ultra60, which is already not quiet :) no, that's the baby-mulching machine in the background landry, i think it would matter less if you were not running this machine in a closet too bad i have not been able to figure out how to drive the fans on this machine. i'd be happy if i could report them, to be fair.
Getting the IP26 kernel to run turned out to be challenging. I eventually
reached a state where the machine would run single-user, but processes would
randomly fail with segmentation faults.
I commited the
R8000
support
on september 29th:
Basic R8000 processor support. R8000 processors require MMU-specific code, exception-specific code, clock-specific code, and L1 cache-specific code. L2 cache is per-design, of which only two exist: SGI Power Indigo2 (IP26) and SGI Power Challenge (IP21) and are not covered by this commit. R8000 processors also are 64-bit only processors with 64-bit coprocessor 0 registers, and lack so-called ``compatibility'' memory spaces allowing 32-bit code to run with sign-extended addresses and registers. The intrusive changes are covered by #ifdef CPU_R8000 stanzas. However, trap() is split into a high-level wrapper and a new function, itsa(), responsible for the actual trap servicing (which name couldn't be helped because I'm an incorrigible punster). While an R8000 exception may cause (via trap() ) multiple exceptions to be serviced, non-R8000 processors will always service one exception in trap(), but they are nevertheless affected by this code split.
This was followed with the
IP26
support
minutes later.
Work in progress support for the Power Indigo2 R8000 system (IP26). This is basically an IP22 system (R4000 Indigo2) with the ECC memory board of IP28, and a so-called ``streaming'' L2 cache. IP26 kernels currently boot single-user, but don't live long; I am suspecting a bug in the tcc cache routines, but am currently not able to find it (come to think of it, my understanding of how this cache works could be wrong, and of course there is no documentation for it but what can be gathered from IRIX'comments and defines). Hopefully this situation will improve in the near future; in the meantime I am commiting this as `work in progress' to make sure this code doesn't get lost.
fall 2012 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP6, IP10 | Personal Iris 4D/2x | complete distribution no X server |
||
| IP12 | Indigo (R3000) | complete distribution | ||
| IP20 | Indigo (R4000) | complete distribution | complete distribution no X server |
|
| IP22 | Indigo2 | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only |
complete distribution no X server |
| IP24 | Indy | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only |
complete distribution no X server |
| IP26 | POWER Indigo2 R8000 | code in the public source tree no distribution yet |
||
| IP27 | Origin 200, Origin 2000 | complete distribution | complete distribution no SMP |
|
| IP28 | POWER Indigo2 R10000 | same as IP22 | complete distribution no X server |
|
| IP30 | Octane | not-yet-integrated kernel patches X server on Impact only |
complete distribution no X server |
|
| IP32 | O2 | complete distribution no R10000 support |
complete distribution | complete distribution |
| IP35 | Fuel, Origin 300, Origin 350, Origin 3000, Onyx 350, Onyx 4, Tezro | complete distribution no SMP no X server Origin 300 and 3000 not supported |
2014, OpenBSD
Being unable to figure out what was wrong with my code on IP26 was quite a
setback to me, and I didn’t do much work on sgi after that.
Some time ago, I had swapped Matthieu Herrb‘s O2 mainboard with a
matching mainboard (same processor and memory), in order to be able to update
its older PROM to a version supporting the RM5200 processor, so that he could
replace the 180MHz R5000 processor module with a 300MHz RM5200 module. The PROM
update can only be done from a fairly recent IRIX system, using
flashinst -T -y /usr/cpu/firmware/IP32prom.img
I had brought his mainboard to a friend who was running IRIX on his sgi systems
and had accepted to let me perform the update, so we put Matthieu’s mainboard in
my friend’s O2, started IRIX, and ran flashinst.
I brought the updated mainboard, with the new processor, to Matthieu
mid-december 2013. Shortly afterwards, he reported to me that, when he would try
to halt or reboot the machine, it would shutdown as usual, but then hit a kernel
panic due to an invalid memory access.
The obvious culprit was the PROM update, and then I realized none of the
O2
mainboards I had around was running the latest 4.18 PROM I had flashed for
Matthieu. I thus visited my IRIX friend again on january 17th, to flash another
O2 mainboard.
I went back home, put the mainboard back in its chassis, booted OpenBSD, logged
in, issued “shutdown -r now”, and hit the same kernel panic as Matthieu.
A few hours of debug showed that the ARCBios-provided function pointers for
halt, reboot and powerdown had wrong addresses! The OpenBSD kernel would trust
these pointers but end up jumping in the middle of an ARCBios routine. No wonder
this ended up with invalid memory accesses.
I added a
crude
workaround,
checking if the ARCBios function pointer values
would match the 4.18 values, and update their values in this case.
For some reason (lack of testing being my #1 candidate), IP32 PROM version 4.18 ends up with the CKSEG1 function pointer values being wrong. All of them. Wrong, as in, off-by-0x60. Of course, invoking the advertized function pointers leads to interesting results, from bogus panics to invalid pointer dereferences. Attempt to identify this revision by: - checking that all five function pointer values, as set up by bios_ident(), match the 4.18 bogus values; - instructions at said pointer match the 4.18 values. If the test is positive, then the pointer values are replaced with the correct values. This allows O2 systems with 4.18 PROM to correctly powerdown, reboot and halt. Found the hard way by matthieu@ after a PROM upgrade, verified on a second system by me.
2015, OpenBSD
In september 2015, Naruaki Etomi, who had an R8000 Indigo2
system, tried to run OpenBSD on it, noticed binaries would not work reliably,
and investigated why. He sent a
patch
allowing him to boot multiuser, to the OpenBSD/sgi mailinglist.
Although his patch was not completely correct, it made me realize my bug. In a
set of assembler instructions computing the position of a page table entry, I
was incorrectly clearing one bit too much. This caused half of the virtual
addresses of a process to end up aliasing lower addresses instead of pointing to
the right memory pages, causing memory corruption. In fact, it was a little
miracle that I had been able to reach single-user mode (it’s a good thing
OpenBSD’s /bin/sh is reasonably small, I doubt I would have had the
same outcome, had the shell been bash or zsh.)
I quickly
commited
a fix
for the problem, and could enjoy my R8000
Indigo2
booting multiuser!
I could then enable the build of IP26 kernels and boot blocks in the OpenBSD/sgi
snapshots, on september 27th.
Visa Hankala then spent almost the whole month of december to get
multiprocessor operation on the IP27 and IP35 systems, finally allowing Theo’s
Origin 350 machine to use its four processors. He also switched the pmap
module to use 3-level page tables, finally allowing the 64-bit userland to use
up to one terabyte of virtual memory, instead of the 2GB it was still
constrained with.
fall 2015 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP6, IP10 | Personal Iris 4D/2x | complete distribution no X server |
||
| IP12 | Indigo (R3000) | complete distribution | ||
| IP20 | Indigo (R4000) | complete distribution | complete distribution no X server |
|
| IP22 | Indigo2 | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only |
complete distribution no X server |
| IP24 | Indy | complete distribution XL (newport) graphics only |
complete distribution XL (newport) graphics only |
complete distribution no X server |
| IP26 | POWER Indigo2 R8000 | complete distribution no X server |
||
| IP27 | Origin 200, Origin 2000 | complete distribution | complete distribution | |
| IP28 | POWER Indigo2 R10000 | same as IP22 | complete distribution no X server |
|
| IP30 | Octane | not-yet-integrated patches X server on Impact only |
complete distribution no X server |
|
| IP32 | O2 | complete distribution no R10000 support |
complete distribution | complete distribution |
| IP35 | Fuel, Origin 300, Origin 350, Origin 3000, Onyx 350, Onyx 4, Tezro | complete distribution no X server Origin 300 and 3000 not supported |
2018, OpenBSD
Although I had not been active in OpenBSD for three years, I was still tinkering
from time to time with my SGI systems.
While playing with O2 machines, I had noticed that, in some cases,
the network interface would operate very slowly, but had no idea why.
Eventually, I started to investigate, and gathered some clues. This ended up
with a
fix
sent to the OpenBSD tech mailinglist.
Date: Sat, 8 Dec 2018 17:32:41 +0000 From: Miod Vallat To: tech@openbsd.org Subject: SGI O2 mec(4) cold boot issue (and workaround) I have noticed, for a while, that my O2 systems were horribly slow during installs or upgrades, when fetching sets from the network (28 *minutes* to fetch base64.tgz). At first, I thought this was a bsd.rd specific bug, but couldn't find anything obvious. After gathering enough data, I found out that the problem only occurs on a cold boot. After a reboot, the network performance is as good as it can be. That would explain why I would only notice it during upgrades. I also noticed that, on a warm boot, the dmesg would show: mec0 at macebus0 base 0x00280000 irq 3: MAC-110 rev 1, address 08:00:69:0e:bf:a1 nsphy0 at mec0 phy 8: DP83840 10/100 PHY, rev. 1 but on cold boots, it would show: mec0 at macebus0 base 0x00280000 irq 3: MAC-110 rev 1, address 08:00:69:0e:bf:a1 nsphy0 at mec0 phy 10: DP83840 10/100 PHY, rev. 1 Note that, in these cases, the phy seems to attach to a different address. In these cases, after booting, "ifconfig mec0" would show: mec0: flags=8843mtu 1500 lladdr 08:00:69:0e:bf:a1 llprio 3 media: Ethernet autoselect status: active inet 10.0.1.193 netmask 0xff000000 broadcast 10.255.255.255 while one would expect the "media" line to be similar to: media: Ethernet autoselect (100baseTX full-duplex) Investigating further, it seems that, after a cold boot, the MII bus takes some time to initialize; the phy does not answer to address 8 but to a larger address (10 or 11), then, after being reset, to its correct address of 8. So the kernel would discover the phy at a wrong address, attach it, and after it gets reset, reading from the phy at the wrong address would return either all bits clear or all bits set, confusing the link speed logic without any way to recover. What I tried but did not work: - invoking mii_attach() twice in the mec driver. This would attach nsphy twice, once at the wrong address, then once at the correct address, but the first (wrong) attachment would be preferred. - adding a one second delay between the Ethernet interface reset and mii_attach(). This would work most of the time, but not always. What I tried and works: - the first time the interface is reset, the mii bus is walked and all phys found on it are reset. Thus, by the time mii_attach() runs and walks the bus again, the phy will answer at the right address. The diff below implements this (last chunk of if_mec.c), and also cleans the mii read/write routines a bit (all the other chunks). Tested on three different R5K family O2 systems, which have all been exposing that problem on cold boot.
Visa Hankala commited that fix (as well as a few other minor diffs I had) for
me.
2021, OpenBSD
All this work was short-lived, as Theo de Raadt decided to pull the plug
on OpenBSD/sgi in 2019.
Date: Mon, 26 Aug 2019 09:34:46 -0600 From: Theo de Raadt To: Miod Vallat Subject: sgi I'm going to terminate sgi base builds. It's been fun. But the disk controller hang makes it pretty difficult for me to do repeated builds. It happens during intense-write, about every 4th-8th release build, towards the end when installing to destdir. But worse, it also happens very often when relinking libc. The octeon port is working a fair bit better these days, I've done 80 builds a in a row with visa's latest pmap change. It is fairly fast. Also, I've come to an agreement with Rhino to buy some of their machines with have M.2 SATA slots, so we'll be able to stand up a pkg cluster. For loongson, things are looking dire. The addition of clang there has been dismal. longsoon takes longer to build than landisk. I'll keep carrying on with as many test canary architectures as possible, but each time the sgi eats itself there is a 10% chance it eats it's filesystem quite badly....
Despite this mail, OpenBSD/sgi release builds were still performed until the
6.9 release in may 2021, after which the port was removed from the tree.
Note that, despite the end of the OpenBSD/sgi port, a kind soul is still
keeping the
sgi kernels up to date (the userland can be obtained from the
OpenBSD/octeon distribution
sets, as Octeon share the same endianness as SGI processors.)
fall 2021 status
| SGI model | common name | Linux | NetBSD | OpenBSD |
| IP6, IP10 | Personal Iris 4D/2x | complete distribution no X server |
||
| IP12 | Indigo (R3000) | complete distribution | ||
| IP20 | Indigo (R4000) | complete distribution | no more… | |
| IP22 | Indigo2 | complete distribution? XL (newport) graphics only |
complete distribution XL (newport) graphics only |
no more… |
| IP24 | Indy | complete distribution? XL (newport) graphics only |
complete distribution XL (newport) graphics only |
no more… |
| IP26 | POWER Indigo2 R8000 | no more… | ||
| IP27 | Origin 200, Origin 2000 | complete distribution? | no more… | |
| IP28 | POWER Indigo2 R10000 | same as IP22 | no more… | |
| IP30 | Octane | complete distribution? X server on Impact only |
no more… | |
| IP32 | O2 | complete distribution? no R10000 support |
complete distribution | no more… |
| IP35 | Fuel, Origin 300, Origin 350, Origin 3000, Onyx 350, Onyx 4, Tezro | no more… |
(to be fair, I am not sure there were still Linux distributions supporting
SGI hardware in 2021 – the Gentoo efforts did not appear to
produce anything,
and Debian
dropped
all its mips ports
in 2019, the year the Octane support had finally been merged into the
linux-mips tree…)
[go back to the previous part]
That has been quite an eventful story. What strikes me the most, in these 20+
years, is that, although SGI hardware was appealing to many kernel developers,
and became affordable in the 2000’s, most of the work towards supporting these
systems have almost often been one-man efforts: in Linux, both the O2 and the
Octane work were done each by a single person; this has also be the case in
NetBSD for most of the SGI families it runs on.
OpenBSD is not much better in that regard, as I have done quite a significant
part of the work once Per Fogelström put the port on good tracks (especially
with a 64-bit kernel from almost the beginning). Of course, I am thankful to
everyone who helped, and glad they did! I have some regrets, such as not trying
much to get the X server to run on more frame buffers (Newport and Impact could
have been done in a reasonable amount of time) and not working on audio
support.
Even if that work seems to be lost, it survives in the Attic of the
OpenBSD source tree, and I hope that the various lengthy comments I have put in
these drivers can be used as good documentation, should someone want to tinker
on these SGI systems in the future, regardless of which operating system they
want to run.
OpenBSD will remain forever the first free software operating system to run on
the Fuel, Tezro and Origin 350, as well as on the POWER Indigo2
R8000, despite so many people saying that no other system than IRIX would
even run on that processor, and that’s something noone can ever take from me.
I remain proud of that work, and I hope you have enjoyed this (long) story.
{💬|⚡|🔥} **What’s your take?**
Share your thoughts in the comments below!
#️⃣ **#OpenBSD #SGI #rollercoaster #story**
🕒 **Posted on**: 1772715391
🌟 **Want more?** Click here for more info! 🌟
