My Django webapp deploys fine, however my db connections max out after a few queries to Postgres.
Here is my understanding of my problem:Db connections to Postgres are opened for every query but not closed and this results is a db timeout once max connections is reached.
Heroku documentation for this error, but I think that whatever new number of connections are enabled through an upgrade, those connections will also max out fast. (Or am I wrong?)
pg:killall is not a solution to my problem. The solution must not be manual in this application.
I have used Heroku documentation for Concurrency and DB connection in Django
This SO question is related but answers don't solve my problem
Debug error
FATAL: too many connections for role "DB_username"
Error traceback failure point
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
Running heroku pg:info --app appname in cmd
=== DATABASE_URLPlan: Hobby-devStatus: AvailableConnections: 20/20 <<<<<<<<<<<<<<<<<<<<<PG Version: 12.4Created: 2020-09-11 16:57 UTCData Size: 9.3 MBTables: 15Rows: 85/10000 (In compliance)Fork/Follow: UnsupportedRollback: UnsupportedContinuous Protection: OffAdd-on: postgresql-infinite-00894
Here are the things I have tried:
- setting dj_database_url.config(conn_max_age==0,...)
- Using Pgbouncer to pool connections
- Disabling server side cursors
Settings.py
# PRODUCTION SETTINGSMIDDLEWARE.append('whitenoise.middleware.WhiteNoiseMiddleware')DATABASE_URL = os.environ['DATABASE_URL']DISABLE_SERVER_SIDE_CURSORS = TrueSECRET_KEY = os.environ.get('SECRET_KEY')STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'#HEROKU SETTINGSDATABASES['default'] = dj_database_url.config(conn_max_age=0, ssl_require=True)django_heroku.settings(locals())
Installed pgbouncer buildpack
Procfile
web: bin/start-pgbouncerweb: daphne appname.asgi:application --port $PORT --bind 0.0.0.0 -v2chatworker: python manage.py runworker --settings=appname.settings -v2
This is the only Procfile that worked for me, I've seen people on other forums complain that you can't do 2 web dynos, this works for me, pgbouncer launches successfully at deployment, daphne works too. Using web and web2 crashes app. That being said I am open to trying any and all suggestions.
asgi.py in case you're curious
import osimport djangofrom channels.routing import get_default_applicationos.environ.setdefault("DJANGO_SETTINGS_MODULE", "Stonks.settings")django.setup()application = get_default_application()
The documentation around this is very confusing and other SO/code forums haven't suggested anything I haven't tried. Clarifying whether this is a Django code issue, Heroku config issue, or Daphne issue would be tremendously helpful as well.
consumers.py sample
class StatsConsumer(AsyncConsumer):"""Creates your stats websocket for each connected user""" async def websocket_connect(self, event):"""Connects to HTML and JS websocket for your stats""" await self.send({"type": "websocket.accept" }) player = self.scope['user'] player_bids_list = await self.get_bids(player) result_dict = await self.find_stats(player_bids_list) await self.send({"type": "websocket.send","text": json.dumps(result_dict), }) async def websocket_receive(self, event):"""Is called when yourStats.html pings server for updated data""" player = self.scope['user'] player_bids_list = await self.get_bids(player) result_dict = await self.find_stats(player_bids_list) await self.send({"type": "websocket.send","text": json.dumps(result_dict), }) async def websocket_disconnect(self, event):"""Is called when websocket disconnects""" print("disconnected", event) @database_sync_to_async def get_profile(self, user):"""fetches user profile funds asynchronously""" return Profile.objects.get(user=user).funds @database_sync_to_async def get_bids(self, user):"""fetches the connected user's bids and returns all bids based on id, asynchronously""" userid = User.objects.get(username=user).id result = Bid.objects.filter(username_id=userid) returnlist = [] for i in result: returnlist.append(i.amount) return returnlist