linux: print a better message for "out of memory" errors for kernel filters.

If attempting to set a kernel filter gets ENOMEM, print a message
suggesting that they increase the value of the net.core.optmem_max
sysctl.

See GitHub issue #1089 for an example of a complicated but not too
complicated filter that exceeds the default "other/option memory per
socket" maximum.

Author: guyharris

pcap-linux.c

@@ -4410,7 +4410,18 @@ pcap_setfilter_linux(pcap_t *handle, struct bpf_program *filter)
 			 * the filter for a reason other than "this kernel
 			 * isn't configured to support socket filters.
 			 */
-			if (errno != ENOPROTOOPT && errno != EOPNOTSUPP) {
+			if (errno == ENOMEM) {
+				/*
+				 * Either a kernel memory allocation
+				 * failure occurred, or there's too
+				 * much "other/option memory" allocated
+				 * for this socket.  Suggest that they
+				 * increase the "other/option memory"
+				 * limit.
+				 */
+				fprintf(stderr,
+				    "Warning: Couldn't allocate kernel memory for filter: try increasing net.core.optmem_max with sysctl\n");
+			} else if (errno != ENOPROTOOPT && errno != EOPNOTSUPP) {
 				fprintf(stderr,
 				    "Warning: Kernel filter failed: %s\n",
 					pcap_strerror(errno));