A quick check for a momentary interruption of the internet on FortiGate

0) Check L1 - cabling, interfaces, transceivers, etc..

Check the arp table and interface status:

1) get system arp 

2) diagnose ip arp list 

3) diagnose netlink interface list port5 

4) diagnose hardware deviceinfo nic port5 

Run a packet sniffer to verify whether FGT is sending packets and receives them:

5.1 diagnose sniffer packet port5 'host X.X.X.X' 4 0 a - run this command on a separate terminal at the same time with the 5.2 commands (to stop it enter CTRL+C) 

5.2 execute ping-options source Y.Y.Y.Y  # y.y.y.y is IP address of the existing interfaces

execute ping X.X.X.X  # x.x.x.x is default gateway

execute ping 8.8.8.8 

execute ping fortinet.com 

execute traceroute-options source Y.Y.Y.Y

execute traceroute X.X.X.X 

execute traceroute 8.8.8.8 

FortiGate NP6 protection functions

 sw-np-bandwidth - limits XAUI speed to 2,4,5,6 to use ISF's buffer

- Can prevent IHP/EHP drops

- But may cause drops on ISF (“sw_np_out_drop_pkts”)

gtse-quota - traffic shaper from CPU to NP6 

- Only on platforms with 1 Gbps ports connected directly to NP6, without ISF

HPE – Host Protection Engine

        -Limits the number of packets per second from NP6 to kernel

NP6 not offloading cases

- TCP control flags (SYN, SYN/ACK, FIN, FIN/ACK, RST)

- One icmp/udp request and one response

- FGT wants to generate ICMP message

- An incoming packet should be fragmented  (egress packet can be fragmented by NP)

- Interface flaps

- Policy changes

- Route changes 

- UTM proxy features 

- sflow 

        -Session helper control channel

        - Manually disabled by policy

npu_state_err flags

  

While investigating NPU-relevant issues, the following npu_state_err flags in the session table entry could help you identify the cause of the problem. 

bit 0: npu shaper deny, can't install npu shaper

bit 1: npu accounting deny, can't install npu accounting

bit 2: npu protocol check deny, there is something wrong on protocol state check, can't pass it to npu for processing

bit 3: npu invalid protocol, npu can't handle this protocol

bit 4: npu ipsec tunnel deny,

bit 5: npu header deny, can't set npu header

bit 6: npu macvlan bridge deny and rst_tcp meanings tcp timeout reset

How to troubleshoot ForitOS Security Fabric issues.

=== before troubleshooting and to save your time === 

- please make sure that you are not using VDOMs (VDOMs cannot be enabled when Security Fabric is used) 

- Security Fabric does not support transparent mode.

- You have to have FAZ or FortiManager with the Analyzer feature enabled.

- Security Fabric cannot be enabled on the loopback, IV interfaces.

- Make sure that you have enabled the "FortiTelemetry" service (fabric service, each major FortiOS version might have a different naming) on corresponding interfaces (fabric upstream and downstream interfaces). 

- Make sure there is a connection between FortiGates (you can enable ping on the interface and try to ping).

- Ensure you have specified the valid IP addresses for the security Fabric connection.

=== Troubleshooting=== 

If so, please provide: 

[please create a log file of a session using PuTTY

Run the following commands on the SSH clients:

==== SSH1 (on core FG) ==== 

# show system global 

# show system csf 

# diag sys csf downstream 

# diag sys csf authorization 

# diagnose sys csf neighbor list 

# diagnose test application csfd 1 

# diagnose sniffer packet any 'host x.x.x.x and port 8013' 6 0 a >>>> x.x.x.x = ip address of the new FG there you have enabled "FortiTelemetry" 

--Real time debug-- 

diagnose debug reset 

diagnose debug application csfd -1 / 0 >>> to disable 

diagnose debug enable 

==== SSH2 (on non-core FG) ===== 

# show system global 

# show system csf 

# diag sys csf upstream 

# diagnose test application csfd 1 

# diagnose sniffer packet any 'host x.x.x.x and port 8013' 6 0 a >>>> x.x.x.x = ip address of the core FG there you have enabled "FortiTelemetry" 

--Real time debug-- 

diagnose debug reset 

diagnose debug application csfd -1 / 0 >>> to disable 

diagnose debug enable