Introduction

This troubleshooting guide catalogs common issues, organized by symptom, within three categories: setup, jobs, and data collection. For each symptom, the likely causes and the resolutions to each of those causes are described.

If you are already aware of the cause of an issue and are just looking for the steps for fixing it, see the Common Procedures section.

This guide assumes familiarity with Linux commands. See the manual pages for details about individual Linux commands.

Issues

Setup

Installation fails to complete

During the Impinj ItemSense installation, the script displays "cannot start service itemsense" or "encountered errors while bringing up the project". The script may continue to run, but it eventually exits with a status 1. The most likely cause is a conflict on one of the ports that ItemSense requires, as specified by the port requirements in the Resource Requirements guide. For example, if an Apache web server is running, it is consuming port 80.

Resolution:

  1. Use the following command to see which port is being consumed:
   sudo netstat -tlnp | grep ':80\W\|:3010\W\|:443\W\|:5672\W'


2. Stop the process that is consuming the port and uninstall the package. 3. Go to the directory where you extracted the ItemSense archive file. Rerun the ItemSense installation script in the tarfile directory. For specific instructions, follow the steps beginning at step 5 of the Install Instructions in the Installation Guide.

Error Code 400 Received During Provisioning

A 400 error code received while provisioning could indicate that the request has timed out. This could have a number of possible causes.

Resolution:

  • Check for a firewall blocking the port between ItemSense and the reader agent.
  • Check for typos or other errors in the host name.
  • Check for a reader with a custom password
  • Make sure the provided host name or IP address is correct.
  • Ensure that the reader is able to resolve hostname of ItemSense server using DNS.

Unable to modify reader NTP settings

With Octane Firmware 5.10 and greater, Network Time Protocol (NTP) settings cannot be modified without first disabling the NTP service.

Resolution:

Use the following command to disable the NTP service: 

config network ntp disable

Reader provisioning fails

Networking issues may cause reader provisioning failures. Although ItemSense may be able to communicate with the readers, the readers may not be able to communicate with ItemSense.

Resolution:

  1. Verify connectivity between the readers and ItemSense.
  2. If no networking issues were found, attempt to re-provision the reader.
  3. If you are still unable to provision the reader, contact Impinj support.

ItemSense operates slowly when reporting tags

When the readers are provisioned, the IP address or hostname used to access the ItemSense Management Console (IMC) is written to the reader as the ItemSense address. If this is an IP address or hostname that the reader has trouble connecting to, then the reader will have connectivity issues as it tries to communicate back to the IMC.

Resolution:

  1. When provisioning the readers, be sure to access the IMC using the same IP Address that the readers will be expected to use. This would be the IP address the readers use to communicate back to the IMC.
  2. If the wrong address was used, you may have to re-provision a reader using the correct address.
  3. If the issue persists, contact Impinj support.

Lost administrator password

If the ItemSense administrator password has been lost, it is possible to reset it back to its default value "admindefault".

Resolution:

  1. Call the factory reset API endpoint as specified in the API Documentation

OR

  1. Connect to ItemSense server using the Secure Shell (SSH) protocol.
  2. Use the following command to edit the admin.json file:
   sudo vi ~/containers/itemsense/home/itemsense/data/configstore/users/admin.json
  1. Replace the contents of the file with the following text:
   {"name":"Admin","roles":["Admin"],"visible":true,"passwordHash":"$2a$10$INiYM7LUIq1SDh90qSn6T.tEA1NpIs1/fXhUoOKrQaXGMbKvwlHnu"}
  1. Reboot the ItemSense server, and log in to the IMC using "admindefault" as the password.

Stopping ItemSense produces errors

When ItemSense is stopped, one or more error messages are displayed. For example, you may see a message such as driver "overlay" failed to remove root filesystem. You must not restart ItemSense without addressing the issues.

Resolution:

  1. Address the issues that were indicated by the error messages. If you are unsure how to proceed, contact Impinj Support.
  2. Verify that no containers are running:
   sudo docker ps

Data is lost when restarting ItemSense

When ItemSense is stopped and then restarted, all configuration data is lost.

Resolution:

Contact Impinj Support.

Access to ItemSense is denied when hostname changed

If the ItemSense hostname changes, you will get an "invalid user/password combination" when attempting to access the IMC.

Resolution:

Recreate the docker containers: sudo docker-compose -f itemsense-prod-3.yml down sudo docker volume prune sudo docker-compose -f itemsense-prod-3.yml up -d

Files config.yml and logback.xml are not editable

To make the files config.yml and logback.xml editable, you must first escalate the permissions using either chmod, chmod +x config.yml or sudo, sudo gedit config.yml

Reader Health

Reader is in questionable state of health

The overall health status of a reader is either NOT_RESPONDING or IDLE with last check-in reporting null, as shown in the following screenshot. A reader should check in with ItemSense at 15-second intervals. The most likely cause is that the reader is no longer paired with this instance of ItemSense or the reader is disconnected from the network.

Reader health shows status of NOT_RESPONDING or IDLE with last check-in reporting null"

Resolution:

  1. Determine reader and ItemSense pairing. If the reader is indeed paired to this instance of ItemSense, verify connectivity between the reader and ItemSense; otherwise, proceed to step 2.
  2. Note the IP address of the reader in the reader definition. You will need the reader's IP address to re-pair it to your instance of ItemSense.
  3. Delete the reader by clicking on EDIT at the bottom of the reader definition.
  4. Click on the SCANNER tab on the Readers page.
  5. Select the subnet that includes the IP address of the reader that you deleted.
  6. Click the DISCOVER READERS button. After ItemSense discovers the readers, the reader that you deleted should be listed under Available for registration.
  7. Select the reader and click the REGISTER SELECTED READERS button.

Tag Data Collection

Wrong facility is reported on tag reads

There may be several causes of this issue.

Resolution:

Contact Impinj Support.

Some EPC values do not get reported in PURE_ID, TAG, and UPC12 formats

PURE_ID, TAG, and UPC12 require that the tags have SGTIN-96 encoding. If not correctly encoded, they return no data.

Resolution:

  • Make sure the tags have SGTIN-96 encoding, or
  • Use the RAW format when running queries.

For more information about EPC formats, see Get Items, Get Item History, or Get Item Threshold Transitions under Items in the ItemSense API documentation.

No tags show up in the Items screen

NTP time-synchronization issues are the most common cause of tag-read issues. Readers must be synchronized with each other and with the ItemSense server. The following situations may prevent this synchronization from happening:

  • Option 42 (DHCP option to use NTP servers) is not enabled in the DHCP client configuration file
  • The DHCP server used by the ItemSense server is different from the DHCP server used by the readers
  • An internet-based NTP server is not accessible and no NTP server is configured on the local network

Network issues or a lack of sufficient disk space may also cause tag-read issues.

Resolution:

  1. Check the job configuration and verify that no filters are preventing tags in the field of view from being read.
  2. Visually inspect the LED lights of the readers to verify that the jobs are running and that they have not been rebooted since the start of the job.
  3. Verify that NTP is working properly:
   ntpq -pn

This command displays information about the peer servers specified in the NTP configuration file. Sample output is shown below. A * or + symbol at the beginning of the line indicates that the peer can be used for synchronization. If no peer is marked by a * or a +, NTP is not operating properly.

Information displayed when NTP is working properly

If NTP is working properly, proceed to the next step; otherwise, set up the ItemSense server as an NTP server and configure the readers to use the ItemSense server as their NTP server, as described in the ItemSense Installation document. 4. Verify connectivity between the readers and ItemSense. 5. Confirm that there is enough disk space on the ItemSense server. See the Server Recommendations section in the ItemSense documentation for resource requirements. 6. Prune the ItemSense logs to reclaim disk space. 7. If problems persist, contact Impinj support.

No tags show up in the Item or Threshold event message output

If jobs are running that should be producing Item or Threshold events but no output is seen on properly configured event brokers, there are several potential causes that warrant investigation:

  1. The job does not have the option set for sending tag events to message queues. Resolution: enable the Data Handling option to "Send tag events to message queues", if using the IMC, or pass a true value for the "reportToMessageQueueEnabled" parameter to the ...control/v1/jobs/start URL if using the API.

  2. The filters on the publisher or message queue are too restrictive and are eliminating all events. Resolution: adjust the filters to be less restrictive, double-check spelling on any EPC prefix or Job Name prefix filter fields.

  3. Parsing on the client side is failing due to an added or changed field (compared to the 2018r2 release of ItemSense). This applies to message queues configured using the ...data/v1/items/queues and ...data/v1/items/queues/threshold URL endpoints. Starting with ItemSense version 2.0.3, Item and Threshold events are published with an additional "jobName" field, and Threshold events have a field that was changed from "dockDoor" to "thresholdId". These changes, particularly for the changed field, are not likely to be handled by client code written to work with the 2018r2 release. Resolution: Update the client code to handle the changed and added fields, or better yet, switch to using the newer publisher and broker APIs to configure the data sources (the APIs under ...configuration/v1/events), as these provide more options and control.

RabbitMQ message queue silently dies after one hour

The RabbitMQ broker will quit after an hour of inactivity (no tag reads), and the connected app will not receive any notification that the connection is broken.

Resolution:

No resolution is necessary — this behavior is by design.

Jobs

Job prematurely stops running or fails to start

If a job stops running before it completes or if a job is unable to start and the IMC displays the following message, the problem may be that there is not enough disk space on the ItemSense server.

Message "Error creating job"

Resolution:

  1. Confirm that there is enough disk space on the ItemSense server. See the Server Requirements section in the ItemSense documentation for resource requirements.
  2. Prune the ItemSense logs to reclaim disk space.
  3. If jobs still do not start, contact Impinj Support.

ItemSense is stopped during a Job

If ItemSense is stopped while a job is running, it should be able to continue to receive data from the reader once it restarts. Data may be lost if reader’s buffer fills before ItemSense is started.

Resolution:

  1. Restart ItemSense to reconnect to the job.
  2. If ItemSense is unable to reconnect, it may be necessary to re-provision the reader.

A reader is stopped, or loses its network connection, during a Job

If the network is temporarily disconnected from a reader while the reader is running a job, ItemSense does not display an error message to indicate that it has lost contact with the reader.

Resolution:

  1. Reconnect the reader and the reads will successfully continue.
  2. If the reader buffer did not overflow, no data was lost.

Common Procedures

Verify connectivity between the readers and ItemSense

Although ItemSense may be able to communicate with the readers, the readers may not be able to communicate with ItemSense. Use the following steps to verify that the readers can communicate with ItemSense.

  1. Log on to the ItemSense server.
  2. Use the ifconfig command to see which network interface is being utilized for connectivity between ItemSense and the readers.

  3. Use the tcpdump command to test for reader-to-ItemSense connectivity issues. For example, if eth1 is the network interface, use the tcpdump command as follows:

    • To test for basic reader-to-ItemSense connectivity, specify port 443:
     tcpdump -i eth1 port 443 -w tcpdumpfile.443.pcap
    • To test reader-agent provisioning, specify port 51505:
     tcpdump -i eth1 port 51505 -w tcpdumpfile.51505.pcap
    • To test SSH commands sent to the reader, specify port 22:
     tcpdump -i eth1 port 22 -w tcpdumpfile.22.pcap
    • To test NTP communication from the reader to ItemSense, specify port 123:
     tcpdump -i eth1 port 123 -w tcpdumpfile.123.pcap

  4. When the tcpdump command fails or completes, close tcpdump and load the resulting .pcap file into your preferred network traffic analyzer.

Prune the ItemSense logs

ItemSense logs are contained within the home directory of the user who installed it. Each ItemSense service will have its own directory within the ItemSense parent directory. Generally, log files are only created when ItemSense is being used. In other words, no logs are created if ItemSense isn't being used. 1. Log into the ItemSense server using the ItemSense installer’s login with an SSH or FTP client. 2. Delete the largest logs from the general ItemSense logs directory.

File path: ~/containers/itemsense/home/itemsense/var/output/logs

  1. Delete the largest logs from the API service ItemSense logs directory.

File path: ~/containers/itemsense/home/itemsense/ItemSenseAPIService/var/output/logs

  1. Delete the largest logs from the coordinator ItemSense logs directory.

File path: ~/containers/itemsense/home/itemsense/Coordinator/var/output/logs

Re-provision a reader

Perform the following steps when a reader needs to be provisioned: 1. Delete the reader configuration from the IMC if it exists. 2. Connect to the reader using SSH. 3. Remove the reader’s CAP file: 

   config image removecap
  1. Close the SSH client, and use the IMC to re-add the reader.

Determine reader and ItemSense pairing

You can determine the instance of ItemSense with which a reader is paired by examining the reader's config.json file. Use the following steps to retrieve this file from a reader.

  1. Connect to the reader via secure shell:
   ssh root@
  1. Once connected, use the config network RShell command to enable FTP on the reader:
   config network ftp enable
  1. Exit out of RShell and connect to the reader via an FTP client:
   ftp
  1. Change to the reader's cust directory, download the config.json file, and exit out of your FTP client. The commands may vary slightly, depending on the FTP client that you use. The following commands are an example:
   cd /
   cd cust
   get config.json
   quit
  1. Display the contents of the reader's config.json file by using the cat command:
   cat config.json

Example output: 

  {"agentid":"xArray-11-3F-5F","apikey":"94a6546f-7e61-4860-b22a-3cf762e7cf10","baseurl":"https://10.200.10.23:443/itemsense/","tlscertfile":"/cust/cert.pem"}

The contents of the baseurl field in the .json file specify the instance of ItemSense that the reader is paired with.