Mailing List Archive

Mike,

I would want to run a 24 BERT test from the CO to the Cisco. Put the
Cisco into remote loopback mode and have Verizon run a BERT test through
that loopback. This will test the Cisco LIU, the cable, the SmartJack
and everything else out to the CO. You need to do this for 24 hours
'cause you don't have a 100% failure rate. If you could reproduce the
eoors all the time you could draw conclusions earlier but as it is you'll
need to run long enough to know that you would have seen the problem.

If that test shows errors then you would want to restest with the loopback
being provided by the SmartJack. If it still shows errors Verizon will
accept responsibility and fix it. If not Cisco will blame your cable or
take resonsibility.

Now as far as the cable goes, what is a "certified cable"? Is it a
certified RJ-48C cable? Or is it an ethernet cable? I don't believe
a lack of sheilding is going to cause this problem however you need
to make sure you have your signals on twisted pairs. RJ-48C uses pins
1&2 and pins 4&5. You must have pins 1&2 use a twisted pair, and the
same for pins 4&5. A flat "silver satin" cable will not work. A cable
where the white-blue/blue-white, etc. pairs are not using the right pins
will not work. A combination of two wires where one is <color> with
white stripes and the other is white with <color> stripes makes up a
twisted pair (e.g. white-orange/orange-white).

99% of the time these sort of problems are caused by cabling/connector
issues.

Next question: where are you deriving timing from? You must be
clocking off the incoming T1 from Verizon. If you are using an internal
clock, or clocking off another span which is not itself timed off
Verizon (or a higher Stratum level) you will have problems.

-Vance

Avaya!!!

-----Original Message-----
From: Mike Armstrong [mailto:mfa@lal.ufl.edu]
Sent: Friday, September 19, 2003 3:56 PM
To: cisco-voip@puck.nether.net
Subject: [cisco-voip] Clicking/popping noise on inbound side of PRI


Warning: I've gone through 4 Cisco engineers at TAC, 3 Verizon
technicians, 2 VWICs, and 2 SmartJacks, with no resolution to this
problem yet, so this may not be easy. I'm looking for out-of-the-box
thinking, unencumbered by reason and conventional troubleshooting
procedures. Have a few beers before working on this.

I've got a 3640/NM-HDV/VWIC-1MFT-T1 connected to an Adtran HTU-R
(Verizon calls it a "SmrtJack") which is the T1 (DS1) interface to an
HDSL circuit between us and the Verizon CO. There is an intermittent
loud clicking/popping noise -- like a Geiger counter -- on the inbound
side only. It was first reported in early August around the time we
upgraded the router
to IOS version 12.2(15)T. Reverting to the previous version --
12.2(13)T -- did not help.

Cisco has replaced the VWIC, even though they didn't think it was the
problem. Verizon replaced the HTU-R even though they didn't think it
was the problem. Both Cisco and Verizon claimed it was my unshielded
patch cable between the VWIC and the HTU-R, so I replaced it with a
certified, shielded cable (10 feet long). None of these replacements
had any effect. In all this time, looking at the diagnostic data
recorded by the HTU-R, I have seen only a few errors of any kind
associated with the Telco circuit, and most of them were apparently
caused by us screwing around with the circuit. During the same time, I
have recorded 6.7M Line Code Violations and 800K Path Code Violations
(hereafter referred to as "errors") on the VWIC.

We all believe that layer 1 timing is the issue. I monitored an open
line for a couple of hours and confirmed that the noise was correlated
with T1 controller errors. I've got "sho controller t1 1/0" data going
back to August 8th, with only a few gaps, in a database keyed to the
15-minute-collection-interval time, and from all this data can tell you:

o We see error rates from 0 to 109,333 per 15-minute interval. o We can
go for hours with 0 errors, then jump to tens of thousands in the next
15-minute interval. o Once they start, the errors tend to continue for a
while, but it's not guaranteed. o There are more errors Saturday,
Sunday, and Monday than Tue-Fri. o There is a broad peak from 2PM-10PM.
o There is no evidence of electrical noise (all our equipment is plugged
into UPS, Telco equipment is line powered). o There is no evidence of
EMF noise in the area, measured crudely with a broadband AM receiver. o
I can't correlate error-free or error-intensive periods with much of
anything.

We had the Telco run loopbacks and BERT tests. Cisco has done the same.
They did it together for an hour and a half last week. I've done the
same. None of us has yet been able to produce similar errors in any test
environment.

Relevant router configs are:

controller T1 1/0
framing esf
linecode b8zs
cablelength short 133
pri-group timeslots 1-20,24
no yellow generation
no yellow detection

...

interface Serial1/0:23
no ip address
no logging event link-status
isdn switch-type primary-ni
isdn incoming-voice voice
(encapsulation HDLC by default)

Any AHA!s out there?

Mike Armstrong
University of Florida CREC
Lake Alfred, FL
863.956.5891

_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

Sorry, I neglected to cc the list when I replied to Scott yesterday; here's
the essence of it:

We use 1-20,24 because we're only paying for 20 channels. I also questioned
this, but TAC didn't raise any
flags. Also, everything worked fine for 8 months, so I'm fairly confident
this isn't it.

Late breaking news... Last night I spent about 1/2 hour of quality time with
the
people who built the SmartJack. It has a control port on it -- an RS-232
connection for a VT-100 to do all sorts of diagnostic stuff. I used it
yesterday
morning to do some loopback testing with no problems. However, there were
no errors this morning either. Tonight, I tried to do the same, with error
rates approaching 100,000 in a 15-minute interval, and the VT-100 stuff was
unusable. The screens were unstable, the SmartJack acted like it was
seeing random input from my laptop, etc. -- a real GIGO situation, but I
wasn't inputting anything, let alone G. The Adtran guy was baffled -- said
he couldn't account for that at all. We disconnected the ground, the
router, and my laptop power supply, leaving the SmartJack connected only to
the Telco line and the laptop, which was now running on battery power
alone. The problem persisted. Mr Adtran was even more baffled -- said if
he was trying to design a SmartJack to do that, he wouldn't know where to
start.

The SmartJack derives its power from the HTU-C at the other end of the span.
Suspecting dirty power, we're eyeing the span power now, but don't know how
to measure/diagnose it. As a
first cut, we decided to ask Verizon to plug in a passive T1 monitor on the
customer side of the SmartJack for a while to see if they see errors there.
They're supposed to show up Monday.
If they see errors, the theory goes, it's their problem. I like your
approach, though. Curiously,
Verizon showed up last week to install another T1, which terminates right
next to this one. Problem is, nobody has any idea who ordered it, or what
it's for. Maybe I'll make use of it.

Mike Armstrong
University of Florida CREC
Lake Alfred, FL
863.956.5891
----- Original Message -----
From: "Vance Shipley" <vances@motivity.ca>
To: "Mike Armstrong" <mfa@lal.ufl.edu>
Cc: <cisco-voip@puck.nether.net>
Sent: Friday, September 19, 2003 6:01 PM
Subject: Re: [cisco-voip] Clicking/popping noise on inbound side of PRI


> Mike,
>
> I would want to run a 24 BERT test from the CO to the Cisco. Put the
> Cisco into remote loopback mode and have Verizon run a BERT test through
> that loopback. This will test the Cisco LIU, the cable, the SmartJack
> and everything else out to the CO. You need to do this for 24 hours
> 'cause you don't have a 100% failure rate. If you could reproduce the
> eoors all the time you could draw conclusions earlier but as it is you'll
> need to run long enough to know that you would have seen the problem.
>
> If that test shows errors then you would want to restest with the loopback
> being provided by the SmartJack. If it still shows errors Verizon will
> accept responsibility and fix it. If not Cisco will blame your cable or
> take resonsibility.
>
> Now as far as the cable goes, what is a "certified cable"? Is it a
> certified RJ-48C cable? Or is it an ethernet cable? I don't believe
> a lack of sheilding is going to cause this problem however you need
> to make sure you have your signals on twisted pairs. RJ-48C uses pins
> 1&2 and pins 4&5. You must have pins 1&2 use a twisted pair, and the
> same for pins 4&5. A flat "silver satin" cable will not work. A cable
> where the white-blue/blue-white, etc. pairs are not using the right pins
> will not work. A combination of two wires where one is <color> with
> white stripes and the other is white with <color> stripes makes up a
> twisted pair (e.g. white-orange/orange-white).
>
> 99% of the time these sort of problems are caused by cabling/connector
> issues.
>
> Next question: where are you deriving timing from? You must be
> clocking off the incoming T1 from Verizon. If you are using an internal
> clock, or clocking off another span which is not itself timed off
> Verizon (or a higher Stratum level) you will have problems.
>
> -Vance
>
>

We can't bring the system down for 24 hours, unfortunately, since that's the
only voice link to the outside world.

Cisco and Verizon worked together for about 2 hours on a Friday afternoon a
week ago doing all sorts of loopbacks and Bert testing -- results were
apparently inconclusive, but Verizon replaced the HTU-R "SmartJack" out of
the goodness of their heart. At the time, I didn't know squat about how
this circuit worked, so I just listened to the experts doing their thing.
However, now that I do know squat about it, I believe there's a giant
loophole in Verizon's diagnostic capabilities in this particular
installation, which I'll mention later.

The "certified cable" is a Cat5E-certifed Ethernet cable; I think it's good
for up to 350MHz or something like that. Pairs are 1-2, 3-6, 4-5, 7-8, so
it should work fine in this application. Hell, given the distance involved,
barbed-wire would probably work. I have a friend who runs 10Mbps Ethernet
over about a 50' run using zip cord.

But I digress; we're deriving timing from the line. Interestingly,
switching to clock source internal has little effect.

I talked with the manufacturer of the HTU-R for a while last night, and
looked more closely at how the Telco monitors the span. Since the circuit
is actually an HDSL link, the main function of the HTU-R is to transform the
HDSL signal (2 loops) into a traditional DS1 for delivery to the customer.
Internal error monitoring is extensive, with logs kept for both loops on the
HDSL side, as well as incoming DS1 data. See anything missing? They do not
monitor or log conditions on the outgoing DS1 -- which is where we're seeing
all our errors.

However, the HTU-R has ports on it to allow passive monitoring of the DS1 on
the customer side of things. Verizon is coming out Monday morning to
install their monitoring equipment and we'll let it cook for however long it
takes. We've logged just under a million errors on the last 6 Mondays
(Monday is our 2nd-cleanest day), so we should be able to produce enough
events to see if the Verizon equipment sees the same errors the Cisco
equipment does. If it does, I think Verizon will admit defeat; if it
doesn't, it's back to Cisco.

As I mentioned in a reply to Scott Voll, I've now got reason to suspect the
problem is at the other end of the span, where an HTU-C (CO side of the HDSL
span) supplies power to the HTU-R via the span. I'll keep you posted.

Mike Armstrong
University of Florida CREC
Lake Alfred, FL
863.956.5891
----- Original Message -----
From: "Vance Shipley" <vances@motivity.ca>
To: "Mike Armstrong" <mfa@lal.ufl.edu>
Cc: <cisco-voip@puck.nether.net>
Sent: Friday, September 19, 2003 6:01 PM
Subject: Re: [cisco-voip] Clicking/popping noise on inbound side of PRI


> Mike,
>
> I would want to run a 24 BERT test from the CO to the Cisco. Put the
> Cisco into remote loopback mode and have Verizon run a BERT test through
> that loopback. This will test the Cisco LIU, the cable, the SmartJack
> and everything else out to the CO. You need to do this for 24 hours
> 'cause you don't have a 100% failure rate. If you could reproduce the
> eoors all the time you could draw conclusions earlier but as it is you'll
> need to run long enough to know that you would have seen the problem.
>
> If that test shows errors then you would want to restest with the loopback
> being provided by the SmartJack. If it still shows errors Verizon will
> accept responsibility and fix it. If not Cisco will blame your cable or
> take resonsibility.
>
> Now as far as the cable goes, what is a "certified cable"? Is it a
> certified RJ-48C cable? Or is it an ethernet cable? I don't believe
> a lack of sheilding is going to cause this problem however you need
> to make sure you have your signals on twisted pairs. RJ-48C uses pins
> 1&2 and pins 4&5. You must have pins 1&2 use a twisted pair, and the
> same for pins 4&5. A flat "silver satin" cable will not work. A cable
> where the white-blue/blue-white, etc. pairs are not using the right pins
> will not work. A combination of two wires where one is <color> with
> white stripes and the other is white with <color> stripes makes up a
> twisted pair (e.g. white-orange/orange-white).
>
> 99% of the time these sort of problems are caused by cabling/connector
> issues.
>
> Next question: where are you deriving timing from? You must be
> clocking off the incoming T1 from Verizon. If you are using an internal
> clock, or clocking off another span which is not itself timed off
> Verizon (or a higher Stratum level) you will have problems.
>
> -Vance
>
>