When using traefik configured for swarm mode, set the label on the service instaed of the container. You do that by moving the labels inside of a deploy block in the compose file:
services:
  site:
    ports:
      - 4000:4000
    image: mywebsite:latest
    deploy:
      labels:
        - traefik.site.port=4000
        - traefik.enable=true
        - traefik.frontend.rule=Host:mydomain.com
    networks: 
      - traefik-net