Apache2::SizeLimit httpd process too big

[kaihendry]

This has something to do with https://github.com/unee-t/bugzilla-customisation/blob/master/start#L27

[Mon Jun 18 06:42:51 2018] (9129) Apache2::SizeLimit httpd process too big, exiting at SIZE=417048 KB SHARE=9544 KB UNSHARED=407504 REQUESTS=82 LIFETIME=1188 s
econds
[Mon Jun 18 06:43:03 2018] (9130) Apache2::SizeLimit httpd process too big, exiting at SIZE=419272 KB SHARE=9644 KB UNSHARED=409628 REQUESTS=72 LIFETIME=1011 s
econds
[Mon Jun 18 06:45:11 2018] (9122) Apache2::SizeLimit httpd process too big, exiting at SIZE=424428 KB SHARE=9504 KB UNSHARED=414924 REQUESTS=192 LIFETIME=2485
seconds
[Mon Jun 18 06:45:18 2018] (9124) Apache2::SizeLimit httpd process too big, exiting at SIZE=411324 KB SHARE=9564 KB UNSHARED=401760 REQUESTS=188 LIFETIME=2456
seconds
[Mon Jun 18 06:53:00 2018] (9134) Apache2::SizeLimit httpd process too big, exiting at SIZE=440412 KB SHARE=9564 KB UNSHARED=430848 REQUESTS=89 LIFETIME=1017 s
econds
[Mon Jun 18 06:54:38 2018] (9155) Apache2::SizeLimit httpd process too big, exiting at SIZE=410844 KB SHARE=9544 KB UNSHARED=401300 REQUESTS=62 LIFETIME=689 se
conds
[Mon Jun 18 06:56:08 2018] (9133) Apache2::SizeLimit httpd process too big, exiting at SIZE=423492 KB SHARE=9564 KB UNSHARED=413928 REQUESTS=102 LIFETIME=1229
seconds
[Mon Jun 18 07:12:15 2018] (9157) Apache2::SizeLimit httpd process too big, exiting at SIZE=412612 KB SHARE=9564 KB UNSHARED=403048 REQUESTS=139 LIFETIME=1606
seconds

I'm going to assume the 400MB limit is being exceeded and needs to be tweaked to be a bit higher.

[kaihendry]

Tbh I am not sure if this is having any effect on the UX, so maybe I won't bother adjusting it.

[kaihendry]

I don't think this is an issue. So closing. Feel free to re-open.

[kaihendry]

Noticed this has come up again
Apache2::SizeLimit httpd process too big, exiting at SIZE=410020 KB SHARE=9656 KB UNSHARED=400364 REQUESTS=107 LIFETIME=1411 seconds

https://github.com/unee-t/bugzilla-customisation/blob/master/start#L30 should be bumped to 800_000 under recommendation from dylan.

However we have a balancing act with memory limits:

jfhearn from #bugzilla irc channel told me if we have a lot of products/components, Bugzilla will bloat in memory usage. I am not sure how to optimise out of here, since the Perl code base is a bit of black box to me.

[franck-boullier]

if we have a lot of products/components, Bugzilla will bloat in memory usage

This will be a growing concern as we expect to have A LOT of products and components as we scale (10s... hopefully 100s of thousands of product/components)

[kaihendry]

Maybe we should go for memory optimized instances like the r4.large to deal with this Bugzilla symptom.

https://aws.amazon.com/ec2/pricing/on-demand/

But at this stage, I don't see it interfering with UX. It just that one of the two instances cut out from time to time.

[franck-boullier]

Maybe we should go for memory optimized instances like the r4.large to deal with this Bugzilla symptom

@kaihendry I'll let you make the call on that one, let me know if/when I can help

[dylanwh]

This is one of the things I really should get back-ported to 5, or hurry up and get 6.0 bugzilla released. c.f. https://dylanwh.tumblr.com/post/147319029597/bmo-memory-leaks

[franck-boullier]

This is one of the things I really should get back-ported to 5

That'll be great @dylanwh. Let me know if there's a way for us to sponsor that somehow so we can make it happen soon

[kaihendry]

Just seen it happening again. Oddly there are no 500 in the logs but the target group IIUC will report them since the instance gets killed AFAICT.

aws --profile uneet-dev logs filter-log-events --log-group-name "bugzilla" --start-time $(date -d "-1 hours" +%s000)