Timewarps, virtual-cirkosis and other twitches


Mills@UDEL.EDU
Tue, 15 Dec 87 14:36:54 EST


Folks,

As I have been heard to mutter from time to time (pun), watching local clocks
reveals all kinds of interesting information about network congestion and
routing flaps. At the end of this message is a summary of clock flaps recorded
over the last couple of weeks at fuzzball host udel2, which is synchronized
via the Network Time Protocol (NTP) to WWVB-clock fuzzball umd1 across 1822
gateway dcn-gw, ARPANET and one or the other X.25 gateways at U Maryland. At
issue are the Maryland gateways, which carry a lot of traffic between the
NSFNET and ARPANET communities and are vulnerable to X.25 virtual-cirkosis,
which occurs when the virtual-circuit demand exceeds the maximum of 64
supported by their ACC 5250 interface and driver.

Under conditions of severe congestion packets may be dropped or delayed for
relatively long periods, which mean relative clock offsets measured by NTP
will become unstable, exceeding even the cope of the sophisticated
median-filtering and smoothing algorithms used by the fuzzballs. When this
happens a timewarp (step-change in local time) occurs, which is then logged
for later analysis. It should be understood that timewarps over noncongested
paths, including most NSFNET, ARPANET and local paths, are extremely rare and,
when they do occur, very serious congestion, delay spikes and/or black holes
are occuring.

Over the last two weeks timewarps between udel2 and umd1 have been occuring
with increasing frequency. Casual ping reveals occasional severe congestion
and packet loss across the Maryland gateways, but relatively smooth sail on
other paths within the NSFNET, ARPANET and local sloughs. The pattern of
congestion suggests that black holes pop up between the gateways and specific
destinations and then disappear after a few minutes. This is exactly the
pattern expected when virtual-cirkosis strikes. Black holes would be expected
to affect low-traffic hosts more than high-traffic ones, which tend to
recapture virtual circuits more quickly.

This problem threatens to become the single most damaging factor affecting
quality service between the NSFNET and ARPANET communities. Warnings about the
problem have been issued on a regular basis since early October, when the
problem first showed up at another gateway. There is no doubt that its
incidence can be reduced by spreading the NSFNET/ARPANET load over more
gateways and, in fact, additional gateways are already in place, but not
configured for this function.

Clearly the ACC 5250 and/or driver must be overhauled to provide either more
circuits or more creative management of the existing number. Solutions for the
driver problem, which involve cache-management and replacement strategies
familiar to any student of computer science, have not been forthcoming. There
is very real need to capture resources, voluntary or coerced, to assist in the
solution.

The data below will give some idea of the incidence of the problem, time of
occurance and severity. It may be possible for other users to correlate these
data with their local logs and trouble reports to see if the pattern matches.
Remember, these data reveal only the most severe disturbances. Many, less
severe disturbances are also affecting connections.

20:37:22 012 ?TRAP-I-Clock reset 4 1130 30-Nov-87
20:46:26 012 ?TRAP-I-Clock reset 4 846 30-Nov-87
21:52:41 012 ?TRAP-I-Clock reset 4 1000 30-Nov-87
22:01:48 012 ?TRAP-I-Clock reset 4 653 30-Nov-87 4 flaps
4:32:12 012 ?TRAP-I-Clock reset 4 920 02-Dec-87
4:55:17 012 ?TRAP-I-Clock reset 4 659 02-Dec-87
5:22:33 012 ?TRAP-I-Clock reset 4 641 02-Dec-87
22:24:29 012 ?TRAP-I-Clock reset 4 992 02-Dec-87
22:33:14 012 ?TRAP-I-Clock reset 4 634 02-Dec-87
23:28:00 012 ?TRAP-I-Clock reset 4 737 02-Dec-87 6 flaps
5:26:20 012 ?TRAP-I-Clock reset 4 637 03-Dec-87
5:28:52 012 ?TRAP-I-Clock reset 4 1200 03-Dec-87
6:33:57 012 ?TRAP-I-Clock reset 4 627 03-Dec-87
6:36:36 012 ?TRAP-I-Clock reset 4 628 03-Dec-87
7:35:26 012 ?TRAP-I-Clock reset 4 1188 03-Dec-87
7:46:40 012 ?TRAP-I-Clock reset 4 647 03-Dec-87
12:46:36 012 ?TRAP-I-Clock reset 4 1088 03-Dec-87
12:55:39 012 ?TRAP-I-Clock reset 4 639 03-Dec-87
16:10:01 012 ?TRAP-I-Clock reset 4 937 03-Dec-87
22:11:06 012 ?TRAP-I-Clock reset 4 753 03-Dec-87
22:29:25 012 ?TRAP-I-Clock reset 4 691 03-Dec-87
22:37:21 012 ?TRAP-I-Clock reset 4 659 03-Dec-87
23:30:55 012 ?TRAP-I-Clock reset 4 717 03-Dec-87 13 flaps
2:35:37 012 ?TRAP-I-Clock reset 4 1212 04-Dec-87
2:35:56 012 ?TRAP-I-Clock reset 4 674 04-Dec-87
3:02:18 012 ?TRAP-I-Clock reset 4 785 04-Dec-87
3:33:11 012 ?TRAP-I-Clock reset 4 873 04-Dec-87
3:35:49 012 ?TRAP-I-Clock reset 4 6527 04-Dec-87
3:35:59 012 ?TRAP-I-Clock reset 4 872 04-Dec-87
3:36:31 012 ?TRAP-I-Clock reset 4 11793 04-Dec-87
3:36:41 012 ?TRAP-I-Clock reset 4 829 04-Dec-87
3:39:03 012 ?TRAP-I-Clock reset 4 810 04-Dec-87
4:48:35 012 ?TRAP-I-Clock reset 4 741 04-Dec-87
4:52:18 014 ?TRAP-I-Clock reset 3 1000 04-Dec-87
7:36:38 012 ?TRAP-I-Clock reset 4 675 04-Dec-87
18:17:08 012 ?TRAP-I-Clock reset 4 680 04-Dec-87
19:22:46 012 ?TRAP-I-Clock reset 4 663 04-Dec-87 14 flaps
1:40:42 012 ?TRAP-I-Clock reset 4 649 05-Dec-87
5:12:26 012 ?TRAP-I-Clock reset 4 660 05-Dec-87
5:19:16 012 ?TRAP-I-Clock reset 4 633 05-Dec-87
17:04:36 012 ?TRAP-I-Clock reset 4 660 05-Dec-87 (Saturday) 4 flaps
21:41:11 012 ?TRAP-I-Clock reset 4 3409 06-Dec-87
21:50:22 012 ?TRAP-I-Clock reset 4 832 06-Dec-87
23:55:00 012 ?TRAP-I-Clock reset 4 2142 06-Dec-87 (Sunday) 3 flaps
0:04:26 012 ?TRAP-I-Clock reset 4 652 07-Dec-87
1:24:14 012 ?TRAP-I-Clock reset 4 3407 07-Dec-87
1:33:22 012 ?TRAP-I-Clock reset 4 646 07-Dec-87
2:38:18 012 ?TRAP-I-Clock reset 4 2308 07-Dec-87
2:47:27 012 ?TRAP-I-Clock reset 4 665 07-Dec-87
3:48:37 012 ?TRAP-I-Clock reset 4 2844 07-Dec-87
3:58:05 012 ?TRAP-I-Clock reset 4 638 07-Dec-87
8:20:53 012 ?TRAP-I-Clock reset 4 3185 07-Dec-87
8:30:38 012 ?TRAP-I-Clock reset 4 622 07-Dec-87
9:20:32 012 ?TRAP-I-Clock reset 4 3310 07-Dec-87
9:29:59 012 ?TRAP-I-Clock reset 4 657 07-Dec-87
17:28:46 012 ?TRAP-I-Clock reset 4 405 07-Dec-87
17:29:18 014 ?TRAP-I-Clock reset 3 1000 07-Dec-87
18:47:11 012 ?TRAP-I-Clock reset 4 1624 07-Dec-87
18:55:58 012 ?TRAP-I-Clock reset 4 656 07-Dec-87
19:30:38 012 ?TRAP-I-Clock reset 4 926 07-Dec-87
20:12:21 012 ?TRAP-I-Clock reset 4 654 07-Dec-87
20:24:29 012 ?TRAP-I-Clock reset 4 1081 07-Dec-87
20:33:43 012 ?TRAP-I-Clock reset 4 636 07-Dec-87 19 flaps
16:08:28 012 ?TRAP-I-Clock reset 4 1547 08-Dec-87
16:16:37 012 ?TRAP-I-Clock reset 4 666 08-Dec-87 2 flaps
3:43:00 012 ?TRAP-I-Clock reset 4 1159 09-Dec-87
3:52:06 012 ?TRAP-I-Clock reset 4 665 09-Dec-87
7:22:50 014 ?TRAP-I-Clock reset 3 1000 09-Dec-87
7:24:07 012 ?TRAP-I-Clock reset 4 1263 09-Dec-87
7:33:04 012 ?TRAP-I-Clock reset 4 658 09-Dec-87
16:16:41 014 ?TRAP-I-Clock reset 3 1000 09-Dec-87
16:37:08 012 ?TRAP-I-Clock reset 4 1234 09-Dec-87
16:46:53 012 ?TRAP-I-Clock reset 4 660 09-Dec-87
17:27:01 012 ?TRAP-I-Clock reset 4 1261 09-Dec-87
17:35:10 012 ?TRAP-I-Clock reset 4 661 09-Dec-87
19:33:53 012 ?TRAP-I-Clock reset 4 1419 09-Dec-87
19:42:07 012 ?TRAP-I-Clock reset 4 967 09-Dec-87
19:52:34 012 ?TRAP-I-Clock reset 4 667 09-Dec-87
20:06:59 012 ?TRAP-I-Clock reset 4 1389 09-Dec-87
20:15:27 012 ?TRAP-I-Clock reset 4 634 09-Dec-87 15 flaps
1:19:09 012 ?TRAP-I-Clock reset 4 1200 10-Dec-87
1:28:19 012 ?TRAP-I-Clock reset 4 662 10-Dec-87
17:05:09 012 ?TRAP-I-Clock reset 4 669 10-Dec-87
21:37:20 012 ?TRAP-I-Clock reset 4 1223 10-Dec-87
21:48:40 012 ?TRAP-I-Clock reset 4 677 10-Dec-87
22:05:13 012 ?TRAP-I-Clock reset 4 3457 10-Dec-87
22:13:55 012 ?TRAP-I-Clock reset 4 673 10-Dec-87 7 flaps
1:59:20 012 ?TRAP-I-Clock reset 4 1668 11-Dec-87
2:07:31 012 ?TRAP-I-Clock reset 4 653 11-Dec-87
2:25:09 012 ?TRAP-I-Clock reset 4 1299 11-Dec-87
2:33:59 012 ?TRAP-I-Clock reset 4 650 11-Dec-87
4:43:39 012 ?TRAP-I-Clock reset 4 2021 11-Dec-87
4:52:44 012 ?TRAP-I-Clock reset 4 917 11-Dec-87
6:19:59 012 ?TRAP-I-Clock reset 4 2515 11-Dec-87
6:28:47 012 ?TRAP-I-Clock reset 4 1038 11-Dec-87 8 flaps
3:12:21 012 ?TRAP-I-Clock reset 4 892 12-Dec-87
3:23:19 012 ?TRAP-I-Clock reset 4 2437 12-Dec-87
3:29:57 012 ?TRAP-I-Clock reset 4 1269 12-Dec-87
3:39:08 012 ?TRAP-I-Clock reset 4 724 12-Dec-87
5:43:07 012 ?TRAP-I-Clock reset 4 629 12-Dec-87
5:49:06 012 ?TRAP-I-Clock reset 4 1134 12-Dec-87
6:11:27 012 ?TRAP-I-Clock reset 4 3019 12-Dec-87
6:11:45 012 ?TRAP-I-Clock reset 4 3020 12-Dec-87
6:18:56 012 ?TRAP-I-Clock reset 4 1251 12-Dec-87
6:27:43 012 ?TRAP-I-Clock reset 4 662 12-Dec-87
9:10:28 012 ?TRAP-I-Clock reset 4 1205 12-Dec-87
9:19:34 012 ?TRAP-I-Clock reset 4 696 12-Dec-87
14:01:31 012 ?TRAP-I-Clock reset 4 1101 12-Dec-87
14:10:35 012 ?TRAP-I-Clock reset 4 629 12-Dec-87
22:00:22 012 ?TRAP-I-Clock reset 4 1047 12-Dec-87
22:10:44 012 ?TRAP-I-Clock reset 4 722 12-Dec-87
23:07:13 012 ?TRAP-I-Clock reset 4 926 12-Dec-87 17 flaps
2:46:49 012 ?TRAP-I-Clock reset 4 1435 13-Dec-87
2:54:56 012 ?TRAP-I-Clock reset 4 647 13-Dec-87
5:20:21 012 ?TRAP-I-Clock reset 4 959 13-Dec-87
5:28:10 012 ?TRAP-I-Clock reset 4 871 13-Dec-87
5:37:37 012 ?TRAP-I-Clock reset 4 1201 13-Dec-87
5:49:14 012 ?TRAP-I-Clock reset 4 644 13-Dec-87
22:59:57 012 ?TRAP-I-Clock reset 4 2039 13-Dec-87
23:13:06 012 ?TRAP-I-Clock reset 4 819 13-Dec-87
23:24:59 012 ?TRAP-I-Clock reset 4 1316 13-Dec-87
23:47:33 012 ?TRAP-I-Clock reset 4 674 13-Dec-87 10 flaps
0:23:00 012 ?TRAP-I-Clock reset 4 1254 14-Dec-87
0:31:48 012 ?TRAP-I-Clock reset 4 1577 14-Dec-87
0:40:18 012 ?TRAP-I-Clock reset 4 669 14-Dec-87
1:34:40 012 ?TRAP-I-Clock reset 4 1375 14-Dec-87
1:43:28 012 ?TRAP-I-Clock reset 4 652 14-Dec-87
2:37:20 012 ?TRAP-I-Clock reset 4 3142 14-Dec-87
2:45:50 012 ?TRAP-I-Clock reset 4 656 14-Dec-87
2:46:09 012 ?TRAP-I-Clock reset 4 657 14-Dec-87
5:07:25 014 ?TRAP-I-Clock reset 3 1000 14-Dec-87
5:21:47 012 ?TRAP-I-Clock reset 4 650 14-Dec-87
15:53:09 012 ?TRAP-I-Clock reset 4 1231 14-Dec-87
16:04:25 012 ?TRAP-I-Clock reset 4 695 14-Dec-87
16:37:40 012 ?TRAP-I-Clock reset 4 1304 14-Dec-87
16:48:06 012 ?TRAP-I-Clock reset 4 889 14-Dec-87
16:59:06 012 ?TRAP-I-Clock reset 4 1855 14-Dec-87
17:22:23 014 ?TRAP-I-Clock reset 3 1000 14-Dec-87
19:49:59 012 ?TRAP-I-Clock reset 4 1486 14-Dec-87
19:58:27 012 ?TRAP-I-Clock reset 4 1000 14-Dec-87 18 flaps
1:22:00 012 ?TRAP-I-Clock reset 4 1508 15-Dec-87
1:29:50 012 ?TRAP-I-Clock reset 4 660 15-Dec-87
2:44:13 012 ?TRAP-I-Clock reset 4 1429 15-Dec-87
2:52:22 012 ?TRAP-I-Clock reset 4 1007 15-Dec-87
15:21:31 012 ?TRAP-I-Clock reset 4 1454 15-Dec-87
15:29:41 012 ?TRAP-I-Clock reset 4 665 15-Dec-87
16:51:20 012 ?TRAP-I-Clock reset 4 1724 15-Dec-87
17:00:28 012 ?TRAP-I-Clock reset 4 1060 15-Dec-87
17:58:35 012 ?TRAP-I-Clock reset 4 1431 15-Dec-87
18:08:36 012 ?TRAP-I-Clock reset 4 2431 15-Dec-87
18:19:03 012 ?TRAP-I-Clock reset 4 1080 15-Dec-87 11 flaps (so far)

Dave



This archive was generated by hypermail 2.0b3 on Thu Mar 09 2000 - 14:40:15 GMT