Using Kstat To Check Your NIC Settings On Solaris 10 Unix

Aloha,

Hope you're having an enjoyable Friday and are looking forward to getting out of the office just as badly as I am. It's a long story, but I can't even begin to explain how close I came to just snapping like a twig and going incendiary on my cubicle, the office, the parking lot, the CostCo down the street and pretty much the entire business district. I swear; one more day of this and I might have ended up on the national news being brought down in a hail of bullets as entire Police departments and SWAT Teams on loan from neighboring states littered the urban landscape with speeding chunks of molten hot metal (killing the 100 or so odd "innocent" bystanders, of course). Thank God it's Friday. I never thought I'd ever say that and not feel like an imbecile, but, there you have it. I'm wasted. Worn out. Done. Through. The world has taken its sweet time this week and pummelled me like a crash-test dummy. One day at a time; that's what they say, right?

Hold the phone... I'm getting a late breaking news flash. It seems that today is actually Tuesday (??) Wait... no, it's Monday, definitely Monday (Dustin Hoffman, Rain Man, Judge Wapner, Jeopardy) and I'm writing Tuesday's copy. Okay... that, yes... that does mean that ...the work week has just begun... Man... that's... just... depressing... I... All right; buck up. Keep repeating: "I need this job. I need this job." Don't question authority, do what you're told, toe the line, don't buck the system, don't pass the buck, because the buck stops here. I guess I'll have to live up to the title of this post now... It appears I'm going to need Friday's paycheck. I can't possibly finance the kind of epic malfeasance I'm going to be burning to unleash, four days from now, on the change in my pocket and the fumes in my gas tank. So, hey :) Let's learn something we may already know :)

BTW (which means By The Way if you're too lazy to type the whole thing out, although the explanation kind of defeats the whole purpose... ;), the preceding two paragraphs were in jest. Seriously. I was only kidding, and I apologize to each and every one of you that came here for the few lines of actual cut-and-paste code and technical writing that follow. Of course, if you're a regular reader, you either enjoyed the above, hated it but couldn't stop reading (like not being able to look away from a car wreck, except, this is a just a bunch of words on a "page" and there aren't any cars involved ;) or just skipped right past it. If you're one of the last bunch, you're not even reading this right now, which means I could use this opportunity to make a few totally-uncalled-for remarks, but I won't. Just this once... On with the show :)

Today's we're going to take a look at using kstat on Solaris 10 ( It's actually available on Solaris 9, as well - I'm sure not sure, off the top of my head, about 8, but I think they were still using "netstat -k") to find out the three things you most often want to know about your network interface cards. Of course, the three things you'll want to be able to check on, arise from three separate problem-areas (which almost always get lumped together to some degree). Basically, you'd never need to know this stuff if your NIC's came, out-of-the-box, all cranking at the highest speed possible, transmitting as efficiently as possible and only engaging in worthwhile speed/transfer/protocol negotiations with upstream/downstream routers. But, hey, sometimes the world's not perfect ;)

1. How to check your NIC's link speed with kstat:

They say they sent you a Gig card, but you could swear it's chunking along at 100 meg. You're probably right (or really really really impatient ;), but it never hurts to know for sure (insofar as you can know anything for sure - BTW, be sure to check out my post-existential-emo-nihilist poetry at PleaseKillMeButDon'tMakeItHurt.com ;). This command line should give you that info (-m is the argument for your NIC driver type - might be ce or something else, and -i is for the instance of that driver. In this instance we're using the device driver bge0. -s is for the statistic you're looking up, if you want to specify that; which we do, for now):

host # kstat -m bge -i 0 -s link_speed|grep link_speed|awk '{print $2}'
1000

And, YES, according to the OS, that NIC (or that port, anyway) is performing to expectations. Running at a gig. Fantastic!

2. But, what about the duplex? This is a legitimate concern. I've actually never seen any card run at 1 gig half duplex, but it "is" possible, or they wouldn't account for it. You'll usually call duplex into question when you're running at 100 meg (on a gig-capable card and network) or 10 meg, given the same circumstances. This is easy to find out, too. Just hire a team of monkeys to hammer the keyboard until they come up with this sequence of characters ;)

host # kstat -m bge -i 0 -s link_duplex|grep link_duplex|awk '{print $2}'
2

Yet again, SUCCESS! If you wanted full duplex, anyway. That's what the 2 stands for. You could also end up with a 1 (half-duplex) or a 0 (which, technically, means your link is down. If you're not connected through serial console, ALOM, or something of that sort and you get this result, your version of kstat is either lying to you or it's hurt and confused ;)

3. And, finally, we should probably figure out if our NIC is set to auto-negotiate. This setting is less "definite" than the other two, since the folks who configure your network may require that your NIC be set to auto-negotiate, or (this happens a lot) train specifically to a determined speed and duplex. Auto-negotiation is always the easiest thing for both parties involved, if it works, but if you need to force your NIC to run at 100 meg full duplex and turn off auto-negotiation in order for your server to run on the network, that's that. You either do it or people start to wonder why they're paying you ;)

host # kstat -m bge -i 0 -s cap_autoneg|grep cap_autoneg|awk '{print $2}'
1

This is either great news or the beginning of a headache that could last weeks and drain you of whatever shattered will you have left ;) If you get a 1 back from this command, auto-negotiation is enabled (In a perfect world, you're Golden) and, if a 0 comes back, you're not doing any auto-negotiation.

If you want to change any of these values, check out this really old post we did on figuring out your NIC's speed and duplex on Solaris (to set the properties, just change the -get option for "ndd" to -set). It'll get you all sorted out :) And, yes, that post is somewhat similar to this one, but it's from November 2007 and we were still all hung up on using "netstat -k" to get our info back then ;)

Keep smiling. Even if it hurts ;)

Cheers,

, Mike

---

tbuskey noted this with regards to the kstat command and link speed attribute. Thanks!

---

This still depends on the hardware. EDITOR NOTE: For this card, tbuskey notes that ifspeed is what you should be looking for!

I have an nge card. kstat -m nge -i 0 | grep link
link_asmpause 0
link_autoneg 1
link_duplex 2
link_pause 0
link_state 1
link_up 1
link_duplex 2
link_state 1
$ kstat -m nge -i 0 | grep speed
bus_speed ifspeed 1000000000
ifspeed 1000000000

---

Discover the Free Ebook that shows you how to make 100% commissions on ClickBank!

---

Please note that this blog accepts comments via email only. See our Mission And Policy Statement for further details.

---

Troubleshooting Veritas Cluster Server LLT Issues On Linux and Unix

Hey There,

Today's post is going to steer away from the Linux and/or Unix Operating Systems just slightly, and look at a problem a lot of folks run into, but have problems diagnosing, when they first set up a Veritas cluster.

Our only assumptions for this post are that Veritas Cluster Server is installed correctly on a two-node farm, everything is set up to failover and switch correctly in the software and no useful information can be obtained via the standard Veritas status commands (or, in other words, the software thinks everything's fine, yet it's reporting that it's not working correctly ;)

Generally, with issues like this one (the software being unable to diagnose its own condition), the best place to start is at the lowest level. So, we'll add the fact that the physical network cabling and connections have been checked to our list of assumptions.

Our next step would be to take a look at the next layer up on the protocol stack, which would be the LLT (low latency transport protocol) layer (which, coincidentally, shares the same level as the MAC, so you may see it referred to, elsewhere, as MAC/LLT, or just MAC, when LLT is actually meant!) This is the base layer at which Veritas controls how it sends its heartbeat signals.

The layer-2 LLT protocol is most commonly associated with the DLPI (all these initials... man. These stand for the Data Link Provider Interface). Which brings us around to the point of this post ;)

Veritas Cluster Server comes with a utility called "dlpiping" that will specifically test device-to-device (basically NIC-to-NIC or MAC-to-MAC) communication at the LLT layer. Note that if you can't find the dlpiping command, it comes standard as a component in the VRTSllt package and is generally placed in /opt/VRTSllt/ by default. If you want to use it without having to type the entire command, you can just add that directory to your PATH environment variable by typing:

host # PATH=$PATH:/opt/VRTSllt;export PATH

In order to use dlpiping to troubleshoot this issue, you'll need to set up a dlpiping server on at least one node in the cluster. Since we only have two nodes in our imaginary cluster, having it on only one node should be perfect.

To set up the dlpiping server on either node, type the following at the command prompt (unless otherwise noted, all of these Veritas-specific commands are in /opt/VRTSllt and all system information returned, by way of example here, is intentionally bogus):

host # getmac /dev/ce:0 <--- This will give use the MAC address of the NIC we want to set the server up on (ce0, in this instance). For this command, even if your device is actually named ce0, eth0, etc, you need to specify it as "device:instance"
/dev/ce:0 00:00:00:FF:FF:FF

Next, you just have to start it up and configure it slightly, like so (Easy peasy; you're done :)

host # dlpiping -s /dev/ce:0

This command runs in the foreground by default. You can background it if you like, but once you start it running on whichever node you start it on, you're better off leaving that system alone so that anything else you do on it can't possibly affect the outcome of your tests. Since our pretend machine's cluster setup is completely down right now anyway, we'll just let it run in the foreground. You can stop the server, at any time, by simply typing a ctl-C:

^C
host #

Now, on every other server in the cluster, you'll need to run the dlpiping client. We only have one other server in our cluster, but you would, theoretically, repeat this process as many times as necessary; once for each client. Note, also, that for the dlpiping server and client setups, you should repeat the setup-and-test process for at least one NIC on every node in the cluster that forms a distinct heartbeat-chain. You can determine which NIC's these are by looking in the /etc/llttab file.

host # dlpiping -c /dev/ce:0 00:00:00:FF:FF:FF <--- This is the exact output from the getmac command we issued on the dlpiping server host.

If everything is okay with that connection, you'll see a response akin to a Solaris ping reply:

0:00:00:FF:FF:FF is alive

If something is wrong, the output is equally simple to decipher:

no response from 00:00:00:FF:FF:FF

Assuming everything is okay, and you still have problems, you should check out the support site for Veritas Cluster Server and see what they recommend you try next (most likely testing the IP layer functionality - ping! ;)

If things don't work out, and you get the error, that's great (assuming you're a glass-half-full kind of person ;) Getting an error at this layer of the stack greatly reduces the possible-root-cause pool and leaves you with only a few options that are worth looking into. And, since we've already verified physical cabling connectivity (no loose or poorly fitted ethernet cabling in any NIC) and traced the cable (so we know NICA-1 is going to NICB-1, as it should), you can be almost certain that the issue is with the quality or type of your ethernet cabling.

For instance, your cable may be physically damaged or improperly pinned-out (assuming you make your own cables and accidentally made a bad one - mass manufacturers make mistakes, too, though). Also, you may be using a standard ethernet cable, where a crossover (or, in some instances, rollover) cable is required. Of course, whenever you run into a seeming dead-end like this, double check your Veritas Cluster main.cf file to make sure that it's not in any way related to a slight error that you may have missed earlier on in the process.

In any event, you are now very close to your solution. You can opt to leave your dlpiping server running for as long as you want. To my knowledge it doesn't cause any latency issues that are noticeable (at least in clusters with a small number of nodes). Once you've done your testing, however, it's also completely useless unless you enjoy running that command a lot ;)

Cheers,

, Mike

Disabling Network Devices in the Solaris Boot PROM

There are quite a few bugs out there on SunSolve, regarding system crashes due to this or that network device failure (or driver issues, depending on how you look at it). Most of the time, the advice is to disable the network device as non-destructively as possible. That can be difficult, given the right circumstances. Of course, I'm talking about the "non-destructive" part. Disabling an interface is easy. Lots of people who don't know the first thing about Solaris can do it, if they bang enough keys and have the proper system privilege ;)

The first thing to do is incredibly obvious. Just take out any references to the network device on the host. Say, if you are using ce0 and ce1 and no longer want to use ce1, you could just do the following and you'd be good from that point on and for future reboots:

ifconfig ce1 down
ifconfig ce1 unplumb
rm /etc/hostname.ce1
vi /etc/hosts <--- Optional, to remove ce1 host name/IP entry
vi /etc/netmasks <--- Optional to remove ce1's netmask setting (unless it's on the same subnet as ce0!)
vi OTHER_FILES <--- Any pertinent files were you may have inserted special route commands, etc, that are no longer applicable

The second thing to look for would be OS operations you could perform (perhaps during boot up). As a "For Instance," certain Sun machines, running certain patch levels, have an issue with the hme0 network device (it's technically a device driver) if it's not connected to the network. Even if you aren't using it. This is somewhat annoying because, if you don't set up the configuration to plumb the network device and bring it up, it shouldn't give you any errors. You should only know hme0 exists by looking in a system file like /etc/path_to_inst. But the hme0 device causes the following error to post constantly during boot up and for a while after:

SUNW,hme0:Parallel detection fault

Working around this bug is fairly simple. In this case (and each case will probably be slightly different - troubleshooting can be long and hard some times), you could run the first two lines below, to stop the activity immediately, while logged in. The second line could be added to /etc/system so that the problem wouldn't recur on reboot. It is strongly recommended to backup, or copy off, /etc/system before changing it, so you can use "boot -a" at the PROM level to boot using your old version if the new one causes your future boots to fail!

ndd -set /dev/hme instance 0
ndd -set /dev/hme adv_autoneg_cap 0
set hme:hme_adv_autoneg_cap=0 >>/etc/system

The third, and most drastic, way you'd go about this is to disable the problematic network device at the Solaris PROM level. Before you bring the machine down, run this (we're still using hme0 as an example and note the output by copying and pasting into notepad, or even writing it down:

grep hme /etc/path_to_inst
"/pci@1f,4000/network@1,1" 0 "hme" <--- This is the device (instance 0 of hme, or hme0) that we want to disable!
"/pci@1f,4000/SUNW,hme@5,1" 1 "hme"

Assuming we've already executed something like "init 0" as the root user, we could do the following to disable hme0 from the PROM "ok" prompt. Note that if you run "show-nets" at the PROM level and see truncated information, use it to compare with the device you have listed from before and use the most similar (just slightly clipped) entry in your future arguments.

If none of the patterns match at all, you may have gotten the wrong info from /etc/path_to_inst or your device tree is screwed up beyond what we're specifically dealing with here today. At this point you should be absolutely certain you know which network device to disable at the PROM level. Now, run the following to make the PROM disable, and Solaris forget all about, hme0:

ok nvedit
0: probe-all install-console banner
1: " /pci@1f,4000/network@1" $delete-device drop
2:
ctl-c <--- Typing the control key and the c key together will break you out of nvedit and put you back at the ok prompt
ok nvstore
ok setenv use-nvramrc? true
use-nvramrc? = true
ok reset-all <--- Make sure you enter a line with "set auto-boot? false" before you run this command, if you want your system to stay at the PROM after it resets.

Now, the hme0 network device should finally be completely disabled at the OS level. Solaris should not even know it exists! You may want to consider doing a reconfigure boot ("init 0" followed by "boot -r" at the PROM "ok" prompt or "reboot -- -r" from the OS -- there are a few more ways to do it, but I digress).

And, to answer the inevitable, and reasonable, question: How can I re-enable my network device on Solaris' PROM once I've disabled it?, here's how:

ok setenv use-nvramrc? false
ok nvedit
0: " /pci@1f,4000/network@1"
delete-device
ctrl-u <--- Hitting the control key and u key together will delete the current line from the nvedit buffer. You only need to do this for the device you previously wanted to ignore.
ctrl-u <--- You usually won't have to type this twice. This is just to demonstrate that you can erase as many lines in the buffer as you want before exiting your nvedit session. In this case, you must delete two lines since the device and delete-device instruction are on separate lines.
ctrl-c
ok nvstore
ok reset-all
ok boot -r

Hopefully this has saved you more headaches than it can potentially cause :)

, Mike

linux unix internet technology

Figuring out your NIC's speed and duplex on Solaris

Here's another one admins and users have to do all time. If there's ever an issue with networking, you need to be able to confidently say that your NIC is up at 1Gb full duplex (or whatever your network admin insists).

The way to check this has changed somewhat in Solaris 10, but the old way to check is still available; although not totally reliable.

For instance, you can use "ndd" in all flavors of Solaris (at least from 2.6 up) to get information from /dev/hme (or whatever your NIC's device driver is). Generally, you would look at the speed and duplex settings using the following commands (slight variation depending on NIC's - e.g. 100Mb hme's don't have values for the 1000 Mb queries)

The following commands are pretty useful, and non-destructive, for any device driver, even though you'll get errors for all the stuff that isn't supported:

/usr/sbin/ndd -set /dev/ce instance 0
/usr/sbin/ndd -get /dev/ce adv_1000fdx_cap
/usr/sbin/ndd -get /dev/ce adv_1000hdx_cap
/usr/sbin/ndd -get /dev/ce adv_100fdx_cap
/usr/sbin/ndd -get /dev/ce adv_100hdx_cap
/usr/sbin/ndd -get /dev/ce adv_10fdx_cap
/usr/sbin/ndd -get /dev/ce adv_10hdx_cap
/usr/sbin/ndd -get /dev/ce adv_autoneg_cap

Of course, replace the "/dev/ce" with your particular driver. The only downside to this hack-and-slash method is that you may see 1's (indicating that the parameter is set) rather than 0's (indicating that the paramet is not set) in more than one place (like in adv_1000fdx_cap , adv_100hdx_cap and adv_autoneg_cap all at once ???)

The best way to do it, in my experience is to use either "netstat -k" (In Solaris up to, and including, version 9) or "kstat -p."

In Solaris 9, assuming the same NIC driver and instance "ce0," you can do the following to find out the status of your NIC:

netstat -k ce|grep 0|egrep 'link_speed|link_dupl'

On Solaris 10, you'd do this:

kstat -p|grep ce0|egrep 'speed|dupl'

Basically, the speed should be 1000 for Gb, 100 for 100Mb, etc. Your duplex is represented numerically as either 0, 1 or 2.

0 = No Duplex (or Down)
1 = Half Duplex
2 = Full Duplex

This output is great to show anyone who doubts your conviction ;)

Take care,

, Mike

linux unix internet technology

What A Night - Changing Solaris Machine Names

Hey there,

Some more possibly boring, but definitely useful, trivia about working on Solaris. Just got off an emergency shift re-purposing a whole bunch of boxes.

It's actually really simple work to change a solaris hosts name. Just need to make sure you hit all the bases.

The files you have to edit are the same for Solaris 8/9 and 10, except for one.

All you need to edit, and run, to change your hostname is:

Edit: /etc/hosts - make your main hostname change here. If possible include an entry for your defaultrouter!
/etc/hostname.NIC (e.g. hostname.ce0) - may be more than one of these depending.
/etc/nodename - the main hostname of your server
/etc/inet/net/*/hosts (Solaris 8/9) - just change the hostname in the hosts files.
/etc/inet/ipnodes (Solaris 10) - change the hostname here
/etc/defaultrouter - you can enter an IP here, but I prefer to enter a hostname and list that in /etc/hosts (above)

Then execute:
hostname NewHostname (as root)
uname -S NewHostname (as root)

And reboot if you like. There's always running sys-unconfig, but it takes out every modification you've ever made. It'll even zero out your resolv.conf file (why???)

Anyway - Enjoy your rest - I have to be back at work in a few minutes ;)

, Mike

linux unix internet technology