Nortel Ip-1140E Phone issues

[Visu]

Hello Mike,

I have issues in upgrading the Nortel Ip-1140E phones, The switch(Opt-61C SG)  was upgraded from Rls 4.50W to 5.50J as part of the Switch upgrade i believe we require to have the Ip-Phones to be upgraded to the current firmware.

While we try to do the phone firmware upgrade in specific to 1140E Ip-Phones from 0625C3C to 0625C6R the phone tries to pick up the software & goes into reboot after that the phone comes back up & says Start  DHCP....

Then after that it keeps hanging after that & does not do anything further i have waited for a while & see that it keeps rebooting after every 4-5mins & does not seem to come up at all.

I have made sure that i have the DHCP scope are set right i have defined options 128 (Nortel-i2004-A) , 191(VLAN-2060.), 224(Nortel-i2004-B) inspite of it the phone does not seem to come up. But when you revert back the firmware from 0625C6R to 0625C3C the phones come up with no issues.

This happens to only Ip-1140E phones only & i have no issues with Ip-1120, 1220 series phones they are all in the current version of 0625C6T.

Yet an another thing that i have noticed in the phone is if i un check the VLan filter settings option in the Ip-1140E phones i am able to upgrade the phone firmware with no issues.

Can you please let me know what would be the default settings for the ip-1140E phones & what are the recommendations to get this issues fixed. Also there are about 1700 Ip-1140E phones that i need to upgrade. Appreciate if you can suggest something with respect to it.

[Michael McNamara]

Hi Visu and welcome to the forums!

I see you posted on my blog too, I'll answer your question here so we can share with the community.

As you are well aware it can be very frustrating troubleshooting these types of problems. The culprit can sometimes be the configuration of the phones themselves. However, let me ask some questions.

Does the upgrade actually complete on the 1140e? Your post seems to indicate that the upgrade completes and then on the following reboot of the phone things stop at "Starting DHCP...".

Do you have access to your DHCP server debug logs? Or can you perform a packet trace (port mirror or span port) of the phone booting up? The DHCP server debug logs and/or the packet trace will usually tell you what's going on and where you should be looking.

You may also be confusing the phone by offering both DHCP options, Nortel-i2004-A, Nortel-i2004-B. As a test I would suggest you remove DHCP option 224 from your DHCP scope and just use DHCP option 128 (Nortel-i2004-A) as a test. You might also want to remove DHCP option 191 and just serve up DHCP option 128 in the data VLAN to see if the phone will boot at least start up even if it's technically in the wrong VLAN (the phone would get an IP address in the data VLAN as opposed to the voice VLAN).

If you are using Nortel switch you also need to disable discard-unregistered-frames on the edge switch ports. This feature is known to cause issues for Nortel IP phones when they attempt to upgrade.

If you want to post your DHCP server debug logs or your packet trace please feel free and I'll try to help you further.

Good Luck!

[Visu]

Mike,

Thanks for the quick reply, I have tried with just DHCP Option 128 & it behaves the same way, Once i push out the new firmware from the Signalling Server & then after that i reboot the phone the phone goes thru its boot cycle & comes back with the Node id & Tn-number which is already registered once it tries connecting to the server it comes up with message on the screen /u/fw/x.25..& after couple of secs the phone reboots.

After this reboot the phone does not come up with usually Nortel logo with black screen but it comes up with Nortel ip-phone 11XX in a small font then it says Start DHCP.... & keeps hanging in there for couple of mins & then it reboots & comes back with the same set of Nortel ip-phone 11XX in a small font then it says Start DHCP.... & keeps hanging then reboots again but does not come up at all.

I have the wire shark captures of the one that is working & not working.

[Michael McNamara]

With your description then we can determine that the IP phone isn't actually upgrading but is failing to obtain and IP address when it's in upgrade mode (small text).

What type/model switch is the IP phone connected to? Let me look at your packet trace and I'll update this post.

Update:

I looked at your packet trace (5c-81_ug_fail1.pcap) and see a few issues... your packet trace doesn't include any 802.1q VLAN tags. If you used a laptop to perform the packet trace you need to configure the NIC to 'not' strip the 802.1q VLAN tags. Have a look at this article on the WireShark wiki for more information.

I also see multiple DHCP offers but I don't see the initial DHCP discover requests from the IP phone, very odd. Where you performing the packet trace at the edge switch port and capturing both Tx and Rx traffic?

In any event in frame 92 you can see that you are returning DHCP options 128 and 191. This is a problem and can confuse some of the phones depending on the firmware version and configuration, you should only return DHCP option 191 in the data VLAN (assuming this is the first DHCP request and this frame is in the data VLAN) there is no need to return DHCP option 128 yet.

In frame 101 we can see the IP phone release the initial DHCP address and then in frame 129 we can see a DHCP offer with DHCP option 128 and the phone appears to boot up properly to me.

Hint: you can right click on the UDP/5000 and UDP/7300 packets and click 'Decode As' and select UNISTIM.

In frame 457 I can see the UNIStim command to upgrade the phone using firmware /u/fw/x25.fw, this packet comes from 10.101.1.20 (Succession Signaling Server?) but in the packet the file server IP address is 119.47.120.50 - does this IP address ring any bells?

In frame 519 I can see the DHCP offer from your DHCP server (again no DHCP Discover? - problem in the packet trace) but the DHCP server is offering both DHCP option 128 and DHCP option 191 which might be confusing the 'upgrade boot code'.

I can also see from the trace that you are using VLAN 2069 for your voice VLAN, however, in your post below you mention VLAN 2060? Is this just a typo in your post?

I see multiple DHCP offers to 00:21:e1:fe:5c:81 so can I assume that you have multiple routers on those VLANs? I'm going to guess that's the case.

So I would remove DHCP option 128 from the DHCP scope in your data VLAN and see how that works.

Good Luck!

[Michael McNamara]

Just another quick note looking over the packet trace again. Your DHCP leases are only 1 day in length. I would strongly recommend you use a minimum of 30 days for your IP phones. There's no need putting all that stress and workload on your DHCP server and IP phones since they'll renew their lease when half the lease time is up, which is every 12 hours in your case.

Good Luck!

[Visu]

Mike,

First of all thanks for taking time to go thru the capture files & analysing it. I am sorry for my typo on the VLan id 2060 instead of 2069.

My node id is 10.101.1.10 & i believe 10.101.1.20 might be the one of sig-servers that would be providing the firmware download files.

I am not too very sure of this IP address is 119.47.120.50 certainly does not ring the bell but i would like to check on this. I will go ahead & remove the Option 128 in DCHP scope on the data V-Lan & will have only DCHP scope 191 & see how that works & will let you know on this.

Thanks,

[Visu]

Mike,  I have tried removing DHCP option 128 in the Data V-Lan & just had only DHCP Opt-191 how ever the phone is able to switch from Data V-Lan to my Voice V-Lan where in i present only Opt-128 in the Voice Vlan the phone is able to connect & does come up.

How ever if i were to push the firmware upgrades to the Phone after the phone reboots & comes back up it tries to get to download the firmware also i see the message as Firmware downloading message in the phone indicating downloading firmware /u/fw/x25.fw after few secs it just reboots again & comes back up with the small font indicating Nortel 110X Manual Configuration.

After few secs i see in the display as Start DHCP..... & stays there for couple of mins & keeps rebooting  & goes into cycles & does not recover back. Also if i were to reboot the phone manually by pulling out the power to it & re feed the power it comes back & then starts downloading the firmware once again.

So i am kind of confused why the phone would behave like that since it initially says it is downloading the Firmware & after few secs it downloads the firmware & reboots & does not comeback up at all but if i were to reboot the phone manually second time then it actually downloads the firmware & upgrades the firmware of the set.

Appreciate if you could let me know what could be getting wrong so that we can try & address it. Also had one more quick question what does the VLan Filter check & uncheck settings do..?? Also i would like to know what is the default settings for the phone with 0625C3C firmware if the VALN filter checked does this cause any unwanted issues like that i have mentioned above??

Please let me know.


Also yet an another quick one i was also reading thru couple of Nortel Docs where in it says that you can set the Signalling Server into Turbo mode i would like to know if it would help in easing up the forcing the phones to download the firmware that i would like to get it pushed for 1140E phones..??

Thanks

[Michael McNamara]

Hi Visu,

I'm not sure I'm going to be able to help much more... the phone is failing to boot properly when it's entering it's upgrade mode.

When you talk about

small font indicating Nortel 110X Manual Configuration

When there is an upgrade available for the 1140e IP phone the phone will reboot into an upgrade mode (used to be call BOOTPC mode on the older i2002/i2004 phones). The phone will boot up, download the code, erase the flash memory, write the flash memory and then reboot again. The 1140e IP phones can take upwardsof 5 minutes to upgrade and reboot.

The problem you seem to have, by your description, is that the phone hangs when it enters upgrade mode trying to gather the proper DHCP information.

As a test I would suggest you disable the Voice VLAN option on one of the phones and create a DHCP scope with DHCP option 128 in that same VLAN that is the PVID, usually the data VLAN.. See if the phone upgrades properly when the Voice VLAN option is disable and not present, I suspect it will.

Outside of that you need to perform additional packet captures (you need the 802.1q VLAN tags in the packet traces) and you need the debug logs from your DHCP server. Short of that I would strongly suggest you contact your voice reseller for support and assistance.

Good Luck!

[Visu]

Wanted to provide you some updates after i contacted the vendor. Got to know from the Vendor that there is known issue with the unistim firmware C3C where in the phone won't upgrade with the V-Lan filter option being checked in the phone.

The solution that was provided was to un-check the V-Lan Filter option & then try upgrading the phone with either with C4I or C4L where in the phone would upgrade with no issues. I did try this out & phones were able to upgrade to any release once the V-Lan Filter option was un-checked.

Though the phone still came up with the small font but was able to get upgraded with no issues which is kind of surprising. Also i was told that with new firmware the phones would be writing the firmware in a new block of memory unlike of C3C firmware which always tries to erase the old block & tries to re-write on the same block which causes the phone to into a kind of hang state.

Also once the phone was upgraded to C4I or C4L the phone did support the extended DHCP scope options -224 where in you can have the ability to instruct the phone thru the DHCP server to apply the V-Lan filter or not where in you need have to touch each of the phone manually unlike of the earlier situation.

Apart from that i was also to told that both DHCP scope options 128 & 224 can co-exist in the Voice V-Lan that would not cause any harm to phone since with the new firmware (Any firmware after C4A series) the phones have the capability to distinguish the regular scope option & also the extended scope option.

Based on the phone firmware the phone automatically selects & has capability to go into extended or regular scope options which was not available in the previous set of releases.
 
We are able to get the phones going with no issues & are planning to upgrade the phones to new firmware release of C7J that is current with unistim code 4.1 that is available in the Nortel support site.

[Michael McNamara]

Thanks for the update Visu!

I suspect I never observed this problem because I don't use DHCP assigned Voice VLANs anymore. I've been used ADAC/LLDP-MED for the past 3 years now with great success. In any event though, great information for anyone else that might stumble over this problem.

Cheers!