Integration Service - OutOfMemory and Connection Pool shut down #125
Description
https://app.bugsnag.com/samanvay-research-and-development-foundation/avni-int/errors/682e6a6ed383ca4578dd3f10 - outofmemory
https://app.bugsnag.com/samanvay-research-and-development-foundation/avni-int/errors/682e700866c04233b8e5f5b4 - subsequently connection pool shut down
Int Service Logs at time of out of memory
May 22 00:06:17 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:17,703 INFO SpringTaskScheduler-13 [goonj.worker.AvniGoonjErrorRecordsWorker] Completed page number: 134 with number of errors: 1, for sync direction: GoonjToAvni
May 22 00:06:17 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:17,703 INFO SpringTaskScheduler-13 [goonj.worker.AvniGoonjErrorRecordsWorker] Starting page number: 135 for sync direction: GoonjToAvni
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:22,322 DEBUG SpringTaskScheduler-13 [goonj.service.TokenService] Token still valid
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:22,480 DEBUG SpringTaskScheduler-13 [goonj.repository.GoonjBaseRepository] Successfully fetched data for resource DemandService/getDemands, response status code is 200 OK
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:22,480 DEBUG SpringTaskScheduler-13 [worker.goonj.DemandEventWorker] Processing demand: name 2021/ACC-32179/UPD/081 || uuid a1CC50000004TNoMAM
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: Input: Thu May 22 00:06:22 UTC 2025 in Coordinated Universal Time
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:22.488 ERROR 1956457 --- [askScheduler-13] o.a.goonj.service.BaseGoonjService : Mapping entry not found for observation field: LastUpdatedDateTime
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:22,489 INFO SpringTaskScheduler-13 [avni.client.AvniHttpClient] POST: /api/subject?version=3
May 22 00:06:22 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:22,489 DEBUG SpringTaskScheduler-13 [avni.client.AvniHttpClient] POST https://app.avniproject.org/api/subject?version=3
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:23.473 WARN 1956457 --- [askScheduler-13] o.a.h.c.protocol.ResponseProcessCookies : Invalid cookie header: "Set-Cookie: auth-token=eyJraWQiOiJqQlZHUU9sTzVKaGIyUWJ1R3NvZlZwTWlEdjFmTlh3bmFscU1xTXo2ZitnPSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiI2ZTVhMjU3ZS1mYTYwLTRmNjYtOWE0ZC1mMmNkZGVjNTUzZWYiLCJlbWFpbF92ZXJpZmllZCI6dHJ1ZSwiaXNzIjoiaHR0cHM6XC9cL2NvZ25pdG8taWRwLmFwLXNvdXRoLTEuYW1hem9uYXdzLmNvbVwvYXAtc291dGgtMV9EVTI3QUhKdloiLCJwaG9uZV9udW1iZXJfdmVyaWZpZWQiOnRydWUsImNvZ25pdG86dXNlcm5hbWUiOiJpbnRlZ3JhdGlvbl9zeXN0ZW1fdXNlckBnb29uaiIsImN1c3RvbTp1c2VyVVVJRCI6IjhhOTA2MTk4LWUwOTYtNDVkZC04NDVlLTY1Y2VhYmU4NTQ1YiIsImF1ZCI6IjFkNnJndml0anNmb29ubGtibTA3dWl2Z21nIiwiZXZlbnRfaWQiOiJlZTc5MDkyZi1lMzczLTRjMjQtYWViMy05NWY5OTNmZTE4ZDgiLCJ0b2tlbl91c2UiOiJpZCIsImF1dGhfdGltZSI6MTc0Nzg3MjMxMCwicGhvbmVfbnVtYmVyIjoiKzkxNzM1ODMzNzY2OSIsImV4cCI6MTc0Nzg3NTkxMCwiaWF0IjoxNzQ3ODcyMzEwLCJlbWFpbCI6ImJldWxhaGVAc2FtYW52YXlmb3VuZGF0aW9uLm9yZyJ9.QlFmo6xWhCY87H3UYGDgA5H3auzmY0Qvqb2oA726Fpi3GGHLxLRUMUQZzWZ7CNXFydoST3DQi4uN8smQKDeTlv6bXktJM-ut4ae0UdVG0ov_RrC6IiX7N8vpp9gklB9AHmw8P2cZBTA-12hR3MGcZip0MwOP7Fug263u-Tu7dUn7wFXDoLYG8AAN-566AFtqXVy7S5QpkjMaW-j_2oveiQYFPWDEfW_VRxVMVi51iLKTosMX42KxWz-nC6N2LVdnHTNRE17MH5O9C-aMsXeMQNczhKJ-FdD7c8A1SRUW8_gXAn2v5M6CQNjWWa3V7g1jOTIrBOtMT_6nmIy17zQcxA; Max-Age=3467; Expires=Thu, 22 May 2025 01:04:09 GMT; Path=/; Secure; HttpOnly; SameSite=Lax". Invalid 'expires' attribute: Thu, 22 May 2025 01:04:09 GMT
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:23,474 ERROR SpringTaskScheduler-13 [avni.client.AvniHttpClient] URI: https://app.avniproject.org/api/subject?version=3, Errored Request body: {"Subject type":"Demand","Registration date":"2025-05-22","observations":{"Target Community":["Migrant workers","Adivasis"],"AccountId":"0012w00000fksxMAAQ","Account name":"Goonj Lalitpur","Dispatch Status":"Dispatched","Type of Disaster":"Covid","Demand Status":null,"DemandId":"a1CC50000004TNoMAM","Number of people":null},"Address map":{"District":"Lalitpur","State":"Uttar Pradesh"},"First name":"2021/ACC-32179/UPD/081","Voided":false,"External ID":"a1CC50000004TNoMAM"}
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:23,475 ERROR SpringTaskScheduler-13 [worker.goonj.GoonjEventWorker] Goonj Demand a1CC50000004TNoMAM could not be synced to Goonj Salesforce.
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "org.avni.server.util.BadRequestError: Answer concept with name=Adivasis not found<LF> at org.avni.server.web.request.api.RequestUtils.lambda$getObsValue$2(RequestUtils.java:69)<LF> at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)<LF> at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)<LF> at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)<LF> at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)<LF> at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)<LF> at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)<LF> at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)<LF> at org.avni.server.web.request.api.RequestUtils.getObsValue(RequestUtils.java:71)<LF> at org.avni.server.web.request.api.RequestUtils.putObservation(RequestUtils.java:55)<LF> at org.avni.server.web.request.api.RequestUtils.createObservations(RequestUtils.java:17)<LF> at org.avni.server.web.api.SubjectApiController.updateSubjectDetails(SubjectApiController.java:237)<LF> at org.avni.server.web.api.SubjectApiController.updateSubjectAndSave(SubjectApiController.java:130)<LF> at org.avni.server.web.api.SubjectApiController.post(SubjectApiController.java:125)<LF> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)<LF> at java.base/java.lang.reflect.Method.invoke(Method.java:580)<LF> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)<LF> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)<LF> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)<LF> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)<LF> at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379)<LF> at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)<LF> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)<LF> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)<LF> at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67)<LF> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)<LF> at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)<LF> at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)<LF> at org.avni.server.web.api.SubjectApiController$$SpringCGLIB$$0.post(<generated>)<LF> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)<LF> at java.base/java.lang.reflect.Method.invoke(Method.java:580)<LF> at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)<LF> at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)<LF> at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)<LF> at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926)<LF> at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831)<LF> at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)<LF> at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)<LF> at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)<LF> at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)<LF> at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)<LF> at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)<LF> at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)<LF> at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.avni.server.framework.sync.MutableRequestFilter.doFilter(MutableRequestFilter.java:35)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.avni.server.framework.security.LimitHostNamesFilter.doFilterInternal(LimitHostNamesFilter.java:34)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.springframework.web.filter.ForwardedHeaderFilter.doFilterInternal(ForwardedHeaderFilter.java:173)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108)<LF> at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)<LF> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117)<LF> at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)<LF> at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)<LF> at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.avni.server.framework.security.AuthenticationFilter.doFilterInternal(AuthenticationFilter.java:73)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)<LF> at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)<LF> at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)<LF> at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)<LF> at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)<LF> at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)<LF> at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)<LF> at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$3(HandlerMappingIntrospector.java:195)<LF> at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)<LF> at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74)<LF> at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:230)<LF> at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362)<LF> at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)<LF> at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)<LF> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)<LF> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)<LF> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)<LF> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)<LF> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)<LF> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)<LF> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)<LF> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)<LF> at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:663)<LF> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)<LF> at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384)<LF> at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)<LF> at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)<LF> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)<LF> at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)<LF> at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)<LF> at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)<LF> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)<LF> at java.base/java.lang.Thread.run(Thread.java:1583)<LF>"
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:101)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:168)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:819)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:777)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:751)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:621)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.avni.client.AvniHttpClient.getResponseEntity(AvniHttpClient.java:56)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.avni.client.AvniHttpClient.post(AvniHttpClient.java:82)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.avni.repository.AvniSubjectRepository.create(AvniSubjectRepository.java:110)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.goonj.worker.goonj.DemandEventWorker.processDemand(DemandEventWorker.java:52)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.goonj.worker.goonj.DemandEventWorker.process(DemandEventWorker.java:40)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.goonj.worker.goonj.DemandEventWorker.processError(DemandEventWorker.java:62)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.goonj.worker.AvniGoonjErrorRecordsWorker.retryErroredEntitySync(AvniGoonjErrorRecordsWorker.java:98)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.goonj.worker.AvniGoonjErrorRecordsWorker.process(AvniGoonjErrorRecordsWorker.java:88)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.goonj.job.AvniGoonjFullErrorJob.execute(AvniGoonjFullErrorJob.java:46)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.avni_integration_service.scheduler.IntegrationJobScheduler.lambda$scheduleGoonj$1(IntegrationJobScheduler.java:102)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
May 22 00:06:23 ip-10-100-1-136 start.sh[1956457]: at java.base/java.lang.Thread.run(Thread.java:840)
May 22 00:06:29 ip-10-100-1-136 start.sh[1956457]: Exception in thread "http-nio-6013-Poller" java.lang.OutOfMemoryError: Java heap space
May 22 00:06:29 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:29.834 WARN 1956457 --- [askScheduler-13] o.h.e.loading.internal.LoadContexts : HHH000100: Fail-safe cleanup (collections) : org.hibernate.engine.loading.internal.CollectionLoadContext@24fda97e<rs=HikariProxyResultSet@690029741 wrapping org.postgresql.jdbc.PgResultSet@254ee021>
May 22 00:06:29 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:29.834 WARN 1956457 --- [askScheduler-13] o.h.e.l.internal.CollectionLoadContext : HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries
May 22 00:06:29 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:06:29.893 ERROR 1956457 --- [askScheduler-13] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task
May 22 00:06:29 ip-10-100-1-136 start.sh[1956457]: java.lang.OutOfMemoryError: Java heap space
May 22 00:08:00 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:08:00,000 INFO SpringTaskScheduler-2 [amrit.job.AvniAmritMainJob] Starting the Amrit entities pull from Avni
May 22 00:08:00 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:08:00,000 INFO SpringTaskScheduler-2 [avni_integration_service.util.HealthCheckService] Health check ping for slug:amrit st>
May 22 00:08:00 ip-10-100-1-136 start.sh[1956457]: 2025-05-22 00:08:00,003 ERROR SpringTaskScheduler-2 [avni_integration_service.util.HealthCheckService] Health check ping failed:
May 22 00:08:00 ip-10-100-1-136 start.sh[1956457]: java.lang.IllegalStateException: Connection pool shut down