Skip to main content

IPv6 Troubleshooting for Residential ISP Helpdesks

RIPE-631
Publication date:
23 Feb 2015
State:
Published
Author
  • Lee Howard, John Jason Brzozowski, David Freedman, Jason Fesler, Tim Chown, Sander Steffann, Chris Grundemann, Jen Linkova, Chris Tuska, Daniel Breuer, Jan Žorž
File(s)
PDF (228.1 KB)

Many thanks to the BCOP Taskforce and the IPv6 Working Group for facilitating and supporting this document.

Ideas, comments and suggestions for improvements? Email: [email protected]

1. What is a BCOP?

A Best Current Operational Practices (BCOP) document describes best current operational practice on a particular topic, as agreed by subject matter experts and periodically reviewed by the Internet community.

2. Summary

This BCOP provides a basic and generic foundation for any user centric helpdesk that deals with IPv6 residential ISP customer connectivity. The focus is on techniques and solutions for the most common IPv6 user connectivity issues. While the focus of this document is on residential ISP networks, enterprise IT helpdesks and other frontline support personnel may find information in this BCOP useful.

Note that this document is also of value to ISPs who have not yet deployed IPv6, as their customers might be experiencing connectivity issues caused by inadvertent (or intentional) use of IPv6 transition technologies such as tunnel brokers, 6to4 or Teredo. The Connectivity Test tool described below will automatically detect the use of such tools and report them to your helpdesk technicians.

3. Background and History

Many network operators who deal directly with end users are concerned about deploying IPv6. A common complaint often sounds something like: “We deployed IPv6 in our network and all our services, but will not roll it out to our residential customers because our helpdesk knows nothing about it so we can't do it.”

In many cases, having a simple and generic document on hand to with troubleshooting tips and common procedures could solve an issue. An operations team could just add their specifics and hand their tailored document over to the helpdesk manager saying “Hey, we implemented IPv6 and here are a few basic things that your people need to understand and know”.

While this document cannot encompass all possible problems, it should provide a solid first step for frontline support personnel.

4. Using This Document - Note for Helpdesk Managers

This document is intended to be a template that can be altered and supplemented with all the necessary individual company specifics so that each company can build the most suitable policy and procedures to follow when an IPv6 issue is detected.

This document relies heavily on the IPv6 Connectivity Test tool at isp.test-ipv6.com, written by Jason Fesler. ISPs should strongly consider running a local mirror of the site so they can provide their own support for it. In those cases, replace the text “isp.test-ipv6.com” with “test-ipv6.example.net/isp” or wherever you host your copy of the site or use both to provide internal and external connection comparisons. An ISP may be able to further customise the test tool to leverage their own OSS systems so that the tool reports customer-specific details, for example. (The reuse agreement for the tool allows local customisation).

Several cases rely on checking settings on the home router. The helpdesk technician will need to be familiar with router configurations on the devices their customers may use, or refer to vendor documentation. Also, the technician may need to determine whether the IPv6 address space used by the customer is allocated from the company (it may not be if the customer is using a tunnel broker, for example) so you will need to supply your helpdesk technicians with a list of valid IPv6 prefixes used by your residential service.

Not all issues can be solved by the guidance provided in this document and, based on the helpdesk structure, instructions to escalate an issue may occur at different times.

5. IPv6 Troubleshooting

While supporting a new technology such as IPv6 may appear daunting, in practice, most connectivity problems are NOT IPv6 related at all. Indeed, calls related to IPv6 are very rare.

5.1 Basic IPv6 Test

Have the user visit isp.test-ipv6.com. If they can't reach it, IPv4 is not working. Test connectivity as usual. If they can reach it, use the reference in Section 6 to interpret the results.

5.2 Test Connectivity

If the user cannot reach the site at all, have the user visit IPv6only version of the connectivity test site at ipv6.testv6.com/helpdesk. If the site is reachable, refer to the corresponding section of this document to interpret the results and follow the troubleshooting steps defined there. If the user can reach neither isp.test-ipv6.com nor ipv6.testv6.com/helpdesk, follow standard procedures and scripts to determine whether there's a physical connectivity problem or another standard complication. If the site is available to the user, continue using this document.

5.3 Test DNS

If IPv4 is working but the page is unavailable, check DNS. Verify if server name can be resolved to an address:

Open a terminal window and run the following command:

  • MacOS/Linux/other Unix system: "dig isp.test-ipv6.com +short"
  • Windows: "nslookup isp.test-ipv6.com"
If this command does not show any IPv4 addresses, the problem is not IPv6 related. Troubleshoot the DNS server issue first and return to this document if the user is still experiencing issues after the DNS problem has been solved.

5.4 Check Home Router

If there is a home router, first determine if IPv6 is configured on the WAN.

  1. Check the configuration of the home router: if an IPv6 address is visible on the WAN port (e.g.,2001:db8:1234:abcd::83/64), but not on the LAN port, troubleshoot with vendor documentation.
  2. If the home router has a working WAN connection make sure that the automatic provisioning on the LAN is correct. Ensure that an IPv6 prefix was delegated to the home router and assigned to a LAN interface(s).
    1. Check that an IPv6 prefix is configured on the LAN interface
    2. Check the configuration of router advertisements on the home router
    3. Check the configuration of the DHCPv6 server on the home router
    4. Check the device for the proper provisioned prefix
    5. Check the device for a matching (set of ) LAN prefix(es)
  3. If the home router is providing the correct information to the LAN, check if the user device is overriding those settings.
    1. Check if the user device has static address settings that don't correspond to the settings on the home router
  4. Have the user rerun the isp.test-ipv6.com test and provide new results.
    1. If not fixed, start the troubleshooting process again
    2. Follow your escalation procedure when needed

 

Note: Enabling IPv6 WAN provisioning often requires a router/modem reboot.

5.5 Escalate

If the previous steps have not identified the problem, escalate to a network engineer, using the information in Appendix C.

6. Explanation of Helpdesk Codes on http://isp.test-ipv6.com

isp.test-ipv6.com reports test results in a very compact way:

Helpdesk code: xx
Summary

Status of IPv4, slow/timeout warnings, ASN, and ISP name
Status of IPv6, slow/timeout warnings, ASN, and ISP name
(Major warnings, such as MTU)
Status of reachability to “other sites”

IPv4 address
IPv6 address

 

Ask the user to read the “helpdesk code” near the top (in blue). Then, find that code in blue below to identify the problem and troubleshoot it.

The list of “helpdesk codes” (from http://test-ipv6.com/faq_helpdesk.html):

 

112 IPv4, plus broken IPv6

4 IPv4 only

4t IPv4 plus Teredo

46 IPv4 + IPv6

46t Dual stack, possible tunnel

624 6to4

64 NAT64

64t NAT64, possible tunnel

6 IPv6 only

 

112 - IPv4, Plus Broken IPv6

Example results (Note: fields in angle brackets are for example purposes only. The real output contains values specific to user's network configuration):

Helpdesk code: 112
IPv4, plus broken IPv6
IPv4: good, <AS65536, CableCo>
IPv6: broken
IPv4 address: <192.0.2.1>

Interpretation: IPv6 network connectivity somewhere between the user and the website is broken. IPv6 connections are timing out instead of succeeding (or failing fast to IPv4). The user experience visiting major websites may be suffering and some applications are completely failing.

Assumption: User has already power cycled home router, modem, and device, as part of your standard troubleshooting procedure.

Action:

      1. Determine whether IPv6 is offered to this customer, based on company documentation. If yes, continue to the next step.

      2. Confirm whether the user's equipment (modem, router) supports IPv6, based on company list of approved devices. Some retail equipment may also support IPv6. A firmware upgrade may be required. A reboot may trigger a firmware upgrade, or your company may have documented processes to upgrade firmware.

      3. Identify the customer's IP addresses.
      • Windows 8.1:
        • Press (windows key) to go to the Start screen
        • Type “cmd” to search for the command prompt
        • Click “Command Prompt” to open the application
        • In the command prompt window, type “ipconfig” and press enter
        • When done reviewing the data, type “exit” and press enter
      • Windows 7, Windows Vista:
        • Click the Start button
        • Type "cmd" to search for the command prompt
        • Click "cmd.exe" to open the application
        • In the command prompt window, type "ipconfig" and press enter
        • When done reviewing the data, type "exit" and press enter
      • Windows XP
        • Click the Start button
        • Click the "run" option
        • Type "cmd" and hit enter
        • In the command prompt window, type "ipconfig /all" and press enter
        • When done reviewing the data, type "exit" and press enter
      • Apple OS X
        • (from http://kb.wisc.edu/helpdesk/page.php?id=9257)
        • Open Terminal
        • Click the Spotlight search icon in the upper-right corner
        • Type "Terminal"
        • Click the Terminal icon from the search results
        • Type "ifconfig | grep inet6" to get the list of IPv6 addresses
        • When done, either type "exit" or click the red button on the terminal window

4. Check the IPv6 addresses.

It is normal for devices to have multiple IPv6 addresses. Ignore the addresses that start with “fe80:” (link-local IPv6 addresses) and the address “::1” (the equivalent of 127.0.0.1 in IPv4). Identify any remaining address(es) and use the following table:

 

All addresses start with [ISP allocated IPv6 space]

Network problem.
If your organisation dynamically assigns IP addresses, then escalate.
If your organisation statically assigns IP addresses, verify that the customer has the proper details as per company documentation for WAN, LAN and default gateway. If the customer confirms that the details are entered as assigned, then escalate.

All addresses start with "fc" or "fd"

"check the router". These are ULA, Unique Local Addresses.
(A future version of this document will offer more ULA advice, as we gain more experience.)

All addresses start with "2002:"

6to4 - disable 6to4 using http://support.microsoft.com/kb/929852 (for example, Microsoft Fix it 50412)

All addresses start with "2001:0:"

Teredo - disable using http://support.microsoft.com/kb/929852 (for example, Microsoft Fix it 50412)

One or more addresses start with 2001:db8: or 2005:123:456:789:

This address is a known invalid address. Encourage the customer to call their router vendor for support. The vendor has incorrectly implemented IPv6, using the prefix reserved for documentation (2001:db8::) or another test prefix (e.g., 2005:123::). It is reasonable to expect a firmware update that will fix the problem. Escalate. If the ISP provided the router, escalate to an engineer to escalate to the router vendor. If the customer provided the router, they will have to contact vendor support themselves. Alternatively, the ISP might provide equipment.

Any other single address

Either the host is manually misconfigured or another device on the customer's network is offering broken IPv6 services (and possibly misconfigured). They will need to possibly seek onsite help.

Multiple addresses: At least one does not belong to the customer's IPv6 address space

It is normal for devices to have multiple IPv6 addresses. However, it's possible that the device is having trouble determining which address to use.
Source address selection problem?
Network problem? Escalate.

Multiple addresses, none match our customer IPv6 address space; but at least one address starts with “2001:0:” or “2002:”

Windows: Disable tunnel interfaces using http://support.microsoft.com/kb/929852 (for example, Microsoft Fix it 50412). If only one address remains, test with test-ipv6.com, and consult this table if problems persist.

Still here?

Escalate. :)

 

Action:

Follow standard procedures and scripts to determine whether there's a physical connectivity problem or other standard complication. If only IPv6 is slow; if the user is using a tunnel, suggest they change their tunnel to a closer location.

4 - IPv4 Only

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 4
IPv4 only

IPv4: Good, <AS65536, CableCo>

IPv4 address: <192.0.2.1>
 

 

Interpretation: User's issue is not caused by IPv6, as there is no IPv6 configured. If Teredo was found and mentioned, we determined it is “safe” (and not actively being used for connections to websites by DNS name).

Action:

      1. Determine whether IPv6 should be provided to the user based on user profile/company documentation.
      2. Confirm whether the user's equipment (modem, router) support IPv6, based on company list of approved devices. Some retail equipment may also support IPv6. A firmware upgrade may be required. A reboot may trigger a firmware upgrade or your company may have documented processes to upgrade firmware.
      3. If the company has processes to configure IPv6 manually, do so, make sure the configured changes are permanent throughout reboots and test again. Most residential ISPs automatically configure IPv6 where available, so in most cases manual configuration is not possible.
      4. Configure IPv6 on the user's device using vendor documentation.

4t - IPv4 Plus Teredo

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 4t
IPv4 plus Teredo

IPv4: Good, <AS65536, CableCo>
IPv6: Good, Teredo, preferred
OtherSites: <40/40 good>

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Interpretation: Teredo was used to provide an IPv6 address and the host was configured to actively take advantage of this service. Any website that has an IPv6 presence will be reached using Teredo instead of native IPv4. Modern operating systems do not prefer these kinds of tunnels by default. Be aware that the user might have a very old operating system or a non-default configuration.

Action:

      1. Have the user disable any automatic tunneling mechanisms that are active.
        Teredo is a protocol that runs on the PC and tries to get IPv6 traffic through a NAT device or firewall. Disable tunnel interfaces using http://support.microsoft.com/kb/929852(for example, Microsoft Fix it 50412).
      2. If user is still experiencing connectivity issues, verify connectivity to http://isp.test-ipv6.com again and follow the troubleshooting steps for the corresponding code.

 

46 - IPv4 + IPv6

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 46
IPv4 + IPv6

IPv4: Good, <AS65536, CableCo>
IPv6: Good, <AS65536, CableCo>
OtherSites: 40/40 good

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Interpretation: This is an example of a healthy IPv4 + IPv6 configuration. Both IPv4 and IPv6 are working, at least to this website.

Action:

None, unless the “OtherSites” test reports any problems. If problems are reported, investigate possible IPv6 routing or peering issues. If the company had a dual stack speed test server, try that to identify performance issues.

46t - Dual Stack, Possible Tunnel

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 46t
IPv4 + IPv6

IPv4: Good, <AS65536, CableCo>
IPv6: Good, <AS64511, AnotherCo>
OtherSites: 40/40 good

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Interpretation: This is an example of a healthy IPv4 + IPv6 configuration. The IPv6 connectivity appears to be announced by a different entity (or at least a different BGP ASN) than IPv4.

Action:

Ask the user to read the IPv4 and IPv6 lines. Ensure that the ASN or company name represented makes sense for your organisation. If the user is using a foreign IPv6 service, consider reprovisioning the user to your own IPv6 service.

624 - 6to4

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 624
6to4

IPv4: Good, <AS65536, CableCo>
IPv6: Good, 6to4, Preferred

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Interpretation: “6to4” was used to provide an IPv6 address and the host was configured to actively take advantage of this service. Any website that has an IPv6 presence will be reached using 6to4 instead of native IPv4. Modern operating systems do not prefer these kinds of tunnels by default. Be aware that the user might have a very old operating system or a non-default configuration.

Action:

      1. Have the user disable any automatic tunneling mechanisms that are active. 6to4 is a protocol that tries to get IPv6 traffic through a public relay, using IPv4. Public 6to4 relays offer no SLA and published studies show approximately 15% failure rates. Windows: Disable tunnel interfaces using http://support.microsoft.com/kb/929852 (for example, Microsoft Fix it 50412.
      2. If IPv6 is desired, configure IPv6 (verify the user has an IPv6 address, and a default route) and test again. If user is still experiencing access issues, follow the troubleshooting steps for the corresponding code returned by http://isp.test-ipv6.com/.

64 - NAT64

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 64
NAT64

IPv4: Good, <AS65536, CableCo>
IPv6: Good, <AS65536, CableCo>
OtherSites: <40/40 good>

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Interpretation: IPv6 is operating as expected. IPv4 works only with “named” connections and websites. Connections to a literal IPv4 address will fail (e.g., http://192.0.2.1). This is probably not an error condition; it is explicitly called out in case of support issues with specific IPv4 applications.

Action:

Typically, escalate. If you need IPv4 to work and your organisation has deployed 464xlat, you may need to ensure that your customer has properly configured their device for your 464xlat service and rerun the test. With a working 464xat, your customer should instead see a helpdesk code of “46” (IPv4 + IPv6). If you do not have 464xlat, escalate.

64t - NAT64, Possible Tunnel

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 64t
NAT64, possible tunnel

IPv4: Good, <AS65536, CableCo>
IPv6: Good, <AS64511, AnotherCo>
OtherSites: <40/40 good>

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Interpretation: IPv6 is operating as expected. IPv4 works only with “named” connections and websites. Connections to a literal IPv4 address will fail (e.g., http://192.0.2.1). This is probably not an error condition, it is explicitly called out in case of support issues with specific IPv4 applications.

Action:

Typically, none. If you need IPv4 to work and your organisation has deployed 464xlat, you may need to ensure that your customer has properly configured their device for your 464xlat service and rerun the test. With a working 464xat, your customer should instead see a helpdesk code of “46” (IPv4 + IPv6). If you do not have 464xlat, escalate.

Interpretation (2): The IPv6 connectivity appears to be announced by a different entity (or at least, a different BGP ASN) than IPv4.

Action:

Ask the user to read the IPv4 and IPv6 lines. Ensure that the ASN or company name represented makes sense for your organisation. If the user is using a foreign IPv6 service, consider reprovisioning the user to your own IPv6 service.

6 - IPv6 Only

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Help desk code: 6
IPv6 only

IPv4: no
IPv6: good, <AS65536, CableCo>

IPv6 address: <2001:db8::1>

Interpretation: Only IPv6 found.

Action:

Refer to the IPv4 helpdesk, IPv6 is working perfectly. User seems to have working IPv6 connectivity at least to some external sites. If they are still experiencing issues, escalate.

Note: IPv6 only users do not trigger the “OtherSites” check. Without IPv4, there is no way to know whether to report an IPv6 reachability problem or to ignore it (due to site down).

Note: IPv6only users will need to use “http://ipv6.test-ipv6.com/helpdesk” or “http://ipv6.test-ipv6.com/isp”.

"slow"

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 46, slow
IPv4 + IPv6, slow

IPv4: SLOW, <AS1234, CableCo>
IPv6: SLOW, <AS1234, CableCo>
OtherSites: <40/40 good>

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>
 

 

Interpretation: IPv4 or IPv6 connections to the testipv6.com site that should have been fast took over five seconds.

In the user's device's network settings, check DNS/name servers. If the servers there have IPv6 addresses, check connectivity to those servers:

      • Windows: Open a cmd window and run "ping [address]"
      • MacOS: Open a terminal window and run "ping6 [address]"

If they are not reachable, configure different servers (preferably IPv4 ones unless you are sure that user is on IPv6-only network).

"mtu" - "Possible MTU issues" Warning

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 46
IPv4 + IPv6

IPv4: SLOW, <AS1234, CableCo>
IPv6: SLOW, <AS1234, CableCo>
WARNING: POSSIBLE MTU ISSUE (in RED)
OtherSites: 40/40 good

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

 

Interpretation: IPv6 MTU issues are typically caused by ICMPv6 filtering at some point along the path. Small requests were fast, large requests were slow (and/or timed out). There are too many possible solutions to be checked.

Action:

This issue requires a deeper understanding of IPv6 protocol and solving of this issue depends on how your helpdesk is organised. If you have a two- or three-tier helpdesk, hand it over to second level support, clearly noting that there are MTU issues. If you have just one-tier helpdesk, hand it over to appropriate escalation department/contact, depending on your escalation policy.

MTU issues are most often caused when Path MTU Discovery (PMTUd) [RFC1981] fails.

What to do? Hand off to Engineering.

        1. Check firewall configuration on user's CPE, look for the blocking of some ICMPv6 (specifically type “2”) either explicitly configured, or implicitly established through lack of rules permitting this. (If ICMPv6 is blocked at some levels DHCPv6 will not work).
        2. Check for ICMPv6 filtering on the user's path through your access and core network.
        3. Follow your escalation procedure when needed.

Educate yourself and your helpdesk team on IPv6.
There are several outstanding resources available to folks new to IPv6:

 

"Site(s) with failed connectivity" Warning

Example results (fields in angle brackets are for example purposes only; the real output will contain values specific to user's network configuration):

Helpdesk code: 46
IPv4 + IPv6

IPv4: Good, <AS1234, CableCo>
IPv6: Good, <AS1234, CableCo>
OtherSites: <39/40> good, <1/40> bad

IPv4 address: <192.0.2.1>
IPv6 address: <2001:db8::1>

Site(s) with failed connectivity (in RED)

test-­ipv6.example.com

http://ipv6.test-­ipv6.example.com/images­nc/knob_valid_green.png

Interpretation: One or more sites are unreachable by the user only when using IPv6.

Action:

Ensure the user is using your network services (not tunnels). If the user is using tunnels, disable the tunnel as described above. If this error condition persists, rerun the test troubleshoot as described in the section for the corresponding code.

7. IPv6 Training for Helpdesk

IPv6 training for helpdesks is available from many IPv6 training companies all around the world or can be tailored to suit your needs (e.g. basic IPv6 training for engineers and technical staff ). When you are rolling out IPv6 in production you need to train all your staff to balance the IPv4 and IPv6 knowledge anyway, so talk to your training agency and ask them to adapt their IPv6 education program and train your helpdesk team.

8. Conclusion

Most connectivity problems are NOT IPv6 related at all.

Use http://isp.test-­ipv6.com/ for automated IPv6 troubleshooting.

Educate yourself and your helpdesk team on IPv6.

9. Operator's Specifics

What your helpdesk should also know and is specific to your company (operator)

Insert here your specifics...

 

 

Appendix A: Acknowledgements

The authors would like to thank everyone from the Internet community (at large) who sent us suggestions and comments over email or through direct conversation with the authors.

Special thanks goes to Jernej Horvat from Amis - the idea for this document came out of a discussion with him about the hurdles that he's facing as an operator that prevent him from enabling IPv6 for all residential customers.

We would also like to thank the RIPE BCOP Taskforce and the RIPE IPv6 Working Group Co-Chairs for accepting this work and guiding it through the community consensus process.

Appendix B. Basic Troubleshooting Flowchart

 

ripe-631 - Flowchart

Appendix C. Collecting Data for Escalation

It is recommended to collect some additional troubleshooting information from a user and provide that info to an escalation engineer.

bsd and generic info:

 

linux:

Windows: