This troubleshooting guide catalogs common issues, organized by symptom, within three categories: setup, jobs, and data collection. For each symptom, the likely causes and the resolutions to each of those causes are described.
If you are already aware of the cause of an issue and are just looking for the steps for fixing it, see the Common Procedures section.
This guide assumes familiarity with Linux commands. See the manual pages for details about individual Linux commands.
Installation fails to complete
During the Impinj ItemSense installation, the script displays "cannot start service itemsense" or "encountered errors while bringing up the project". The script may continue to run, but it eventually exits with a status 1. The most likely cause is a conflict on one of the ports that ItemSense requires, as specified by the port requirements in the Resource Requirements guide. For example, if an Apache web server is running, it is consuming port 80.
- Use the following command to see which port is being consumed:
sudo netstat -tlnp | grep ':80\W\|:3010\W\|:443\W\|:5672\W'
2. Stop the process that is consuming the port and uninstall the package. 3. Go to the directory where you extracted the ItemSense archive file. Rerun the ItemSense installation script in the tarfile directory. For specific instructions, follow the steps beginning at step 5 of the Install Instructions in the Installation Guide.
Error Code 400 Received During Provisioning
A 400 error code received while provisioning could indicate that the request has timed out. This could have a number of possible causes.
- Check for a firewall blocking the port between ItemSense and the reader agent.
- Check for typos or other errors in the host name.
- Check for a reader with a custom password
- Make sure the provided host name or IP address is correct.
- Ensure that the reader is able to resolve hostname of ItemSense server using DNS.
Unable to modify reader NTP settings
With Octane Firmware 5.10 and greater, Network Time Protocol (NTP) settings cannot be modified without first disabling the NTP service.
Use the following command to disable the NTP service:
config network ntp disable
Reader provisioning fails
Networking issues may cause reader provisioning failures. Although ItemSense may be able to communicate with the readers, the readers may not be able to communicate with ItemSense.
- Verify connectivity between the readers and ItemSense.
- If no networking issues were found, attempt to re-provision the reader.
- If you are still unable to provision the reader, contact Impinj support.
ItemSense operates slowly when reporting tags
When the readers are provisioned, the IP address or hostname used to access the ItemSense Management Console (IMC) is written to the reader as the ItemSense address. If this is an IP address or hostname that the reader has trouble connecting to, then the reader will have connectivity issues as it tries to communicate back to the IMC.
- When provisioning the readers, be sure to access the IMC using the same IP Address that the readers will be expected to use. This would be the IP address the readers use to communicate back to the IMC.
- If the wrong address was used, you may have to re-provision a reader using the correct address.
- If the issue persists, contact Impinj support.
Lost administrator password
If the ItemSense administrator password has been lost, it is possible to reset it back to its default value "admindefault".
- Call the factory reset API endpoint as specified in the API Documentation
- Connect to ItemSense server using the Secure Shell (SSH) protocol.
- Use the following command to edit the admin.json file:
sudo vi ~/containers/itemsense/home/itemsense/data/configstore/users/admin.json
- Replace the contents of the file with the following text:
- Reboot the ItemSense server, and log in to the IMC using "admindefault" as the password.
Stopping ItemSense produces errors
When ItemSense is stopped, one or more error messages are displayed. For example, you may see a message such as driver "overlay" failed to remove root filesystem. You must not restart ItemSense without addressing the issues.
- Address the issues that were indicated by the error messages. If you are unsure how to proceed, contact Impinj Support.
- Verify that no containers are running:
sudo docker ps
Data is lost when restarting ItemSense
When ItemSense is stopped and then restarted, all configuration data is lost.
Contact Impinj Support.
Access to ItemSense is denied when hostname changed
If the ItemSense hostname changes, you will get an "invalid user/password combination" when attempting to access the IMC.
Recreate the docker containers:
sudo docker-compose -f itemsense-prod-3.yml down
sudo docker volume prune
sudo docker-compose -f itemsense-prod-3.yml up -d
Files config.yml and logback.xml are not editable
To make the files config.yml and logback.xml editable, you must first escalate the permissions using either chmod,
chmod +x config.yml or sudo,
sudo gedit config.yml
Reader is in questionable state of health
The overall health status of a reader is either NOT_RESPONDING or IDLE with last check-in reporting null, as shown in the following screenshot. A reader should check in with ItemSense at 15-second intervals. The most likely cause is that the reader is no longer paired with this instance of ItemSense or the reader is disconnected from the network.
- Determine reader and ItemSense pairing. If the reader is indeed paired to this instance of ItemSense, verify connectivity between the reader and ItemSense; otherwise, proceed to step 2.
- Note the IP address of the reader in the reader definition. You will need the reader's IP address to re-pair it to your instance of ItemSense.
- Delete the reader by clicking on EDIT at the bottom of the reader definition.
- Click on the SCANNER tab on the Readers page.
- Select the subnet that includes the IP address of the reader that you deleted.
- Click the DISCOVER READERS button. After ItemSense discovers the readers, the reader that you deleted should be listed under Available for registration.
- Select the reader and click the REGISTER SELECTED READERS button.
Tag Data Collection
Wrong facility is reported on tag reads
There may be several causes of this issue.
Contact Impinj Support.
Some EPC values do not get reported in PURE_ID, TAG, and UPC12 formats
PURE_ID, TAG, and UPC12 require that the tags have SGTIN-96 encoding. If not correctly encoded, they return no data.
- Make sure the tags have SGTIN-96 encoding, or
- Use the RAW format when running queries.
For more information about EPC formats, see Get Items, Get Item History, or Get Item Threshold Transitions under Items in the ItemSense API documentation.
No tags show up in the Items screen
NTP time-synchronization issues are the most common cause of tag-read issues. Readers must be synchronized with each other and with the ItemSense server. The following situations may prevent this synchronization from happening:
- Option 42 (DHCP option to use NTP servers) is not enabled in the DHCP client configuration file
- The DHCP server used by the ItemSense server is different from the DHCP server used by the readers
- An internet-based NTP server is not accessible and no NTP server is configured on the local network
Network issues or a lack of sufficient disk space may also cause tag-read issues.
- Check the job configuration and verify that no filters are preventing tags in the field of view from being read.
- Visually inspect the LED lights of the readers to verify that the jobs are running and that they have not been rebooted since the start of the job.
- Verify that NTP is working properly:
This command displays information about the peer servers specified in the NTP configuration file. Sample output is shown below. A
+ symbol at the beginning of the line indicates that the peer can be used for synchronization. If no peer is marked by a
* or a
+, NTP is not operating properly.
If NTP is working properly, proceed to the next step; otherwise, set up the ItemSense server as an NTP server and configure the readers to use the ItemSense server as their NTP server, as described in the ItemSense Installation document. 4. Verify connectivity between the readers and ItemSense. 5. Confirm that there is enough disk space on the ItemSense server. See the Server Recommendations section in the ItemSense documentation for resource requirements. 6. Prune the ItemSense logs to reclaim disk space. 7. If problems persist, contact Impinj support.
No tags show up in the Item or Threshold event message output
If jobs are running that should be producing Item or Threshold events but no output is seen on properly configured event brokers, there are several potential causes that warrant investigation:
The job does not have the option set for sending tag events to message queues. Resolution: enable the Data Handling option to "Send tag events to message queues", if using the IMC, or pass a true value for the "reportToMessageQueueEnabled" parameter to the ...control/v1/jobs/start URL if using the API.
The filters on the publisher or message queue are too restrictive and are eliminating all events. Resolution: adjust the filters to be less restrictive, double-check spelling on any EPC prefix or Job Name prefix filter fields.
Parsing on the client side is failing due to an added or changed field (compared to the 2018r2 release of ItemSense). This applies to message queues configured using the ...data/v1/items/queues and ...data/v1/items/queues/threshold URL endpoints. Starting with ItemSense version 2.0.3, Item and Threshold events are published with an additional "jobName" field, and Threshold events have a field that was changed from "dockDoor" to "thresholdId". These changes, particularly for the changed field, are not likely to be handled by client code written to work with the 2018r2 release. Resolution: Update the client code to handle the changed and added fields, or better yet, switch to using the newer publisher and broker APIs to configure the data sources (the APIs under ...configuration/v1/events), as these provide more options and control.
RabbitMQ message queue silently dies after one hour
The RabbitMQ broker will quit after an hour of inactivity (no tag reads), and the connected app will not receive any notification that the connection is broken.
No resolution is necessary — this behavior is by design.
Job prematurely stops running or fails to start
If a job stops running before it completes or if a job is unable to start and the IMC displays the following message, the problem may be that there is not enough disk space on the ItemSense server.
- Confirm that there is enough disk space on the ItemSense server. See the Server Requirements section in the ItemSense documentation for resource requirements.
- Prune the ItemSense logs to reclaim disk space.
- If jobs still do not start, contact Impinj Support.
ItemSense is stopped during a Job
If ItemSense is stopped while a job is running, it should be able to continue to receive data from the reader once it restarts. Data may be lost if reader’s buffer fills before ItemSense is started.
- Restart ItemSense to reconnect to the job.
- If ItemSense is unable to reconnect, it may be necessary to re-provision the reader.
A reader is stopped, or loses its network connection, during a Job
If the network is temporarily disconnected from a reader while the reader is running a job, ItemSense does not display an error message to indicate that it has lost contact with the reader.
- Reconnect the reader and the reads will successfully continue.
- If the reader buffer did not overflow, no data was lost.
Verify connectivity between the readers and ItemSense
Although ItemSense may be able to communicate with the readers, the readers may not be able to communicate with ItemSense. Use the following steps to verify that the readers can communicate with ItemSense.
- Log on to the ItemSense server.
- Use the
ifconfigcommand to see which network interface is being utilized for connectivity between ItemSense and the readers.
tcpdumpcommand to test for reader-to-ItemSense connectivity issues. For example, if eth1 is the network interface, use the
tcpdumpcommand as follows:
- To test for basic reader-to-ItemSense connectivity, specify port 443:
tcpdump -i eth1 port 443 -w tcpdumpfile.443.pcap
- To test reader-agent provisioning, specify port 51505:
tcpdump -i eth1 port 51505 -w tcpdumpfile.51505.pcap
- To test SSH commands sent to the reader, specify port 22:
tcpdump -i eth1 port 22 -w tcpdumpfile.22.pcap
- To test NTP communication from the reader to ItemSense, specify port 123:
tcpdump -i eth1 port 123 -w tcpdumpfile.123.pcap
tcpdumpcommand fails or completes, close
tcpdumpand load the resulting .pcap file into your preferred network traffic analyzer.
Prune the ItemSense logs
ItemSense logs are contained within the home directory of the user who installed it. Each ItemSense service will have its own directory within the ItemSense parent directory. Generally, log files are only created when ItemSense is being used. In other words, no logs are created if ItemSense isn't being used. 1. Log into the ItemSense server using the ItemSense installer’s login with an SSH or FTP client. 2. Delete the largest logs from the general ItemSense logs directory.
- Delete the largest logs from the API service ItemSense logs directory.
- Delete the largest logs from the coordinator ItemSense logs directory.
Re-provision a reader
Perform the following steps when a reader needs to be provisioned: 1. Delete the reader configuration from the IMC if it exists. 2. Connect to the reader using SSH. 3. Remove the reader’s CAP file:
config image removecap
- Close the SSH client, and use the IMC to re-add the reader.
Determine reader and ItemSense pairing
You can determine the instance of ItemSense with which a reader is paired by examining the reader's config.json file. Use the following steps to retrieve this file from a reader.
- Connect to the reader via secure shell:
ssh root@<reader hostname>
- Once connected, use the config network RShell command to enable FTP on the reader:
config network ftp enable
- Exit out of RShell and connect to the reader via an FTP client:
ftp <reader hostname or reader IP address>
- Change to the reader's cust directory, download the config.json file, and exit out of your FTP client. The commands may vary slightly, depending on the FTP client that you use. The following commands are an example:
cd / cd cust get config.json quit
- Display the contents of the reader's config.json file by using the cat command:
The contents of the baseurl field in the .json file specify the instance of ItemSense that the reader is paired with.