I have tried to duplicate the problem, without success as the following code will testify.
<!doctype html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>Untitled Document</title>
<link rel="stylesheet" href="bootstrap/5/css/bootstrap.min.css" />
<link rel="stylesheet" href="css/style.css" />
<script src="dmxAppConnect/dmxAppConnect.js"></script>
<script src="dmxAppConnect/dmxBootstrap5Modal/dmxBootstrap5Modal.js" defer=""></script>
</head>
<body is="dmx-app" id="test">
<div class="container">
<div class="row">
<div class="col text-end">
<button class="btn btn-primary" type="button" data-bs-toggle="offcanvas" data-bs-target="#offcanvas" aria-controls="offcanvasExample">
Open Offcanvas
</button>
<button id="btn1" class="btn btn-success" dmx-on:click="modal1.show()">Open Modal</button>
</div>
</div>
</div>
<!-- offcanvas -->
<div class="offcanvas offcanvas-start" tabindex="-1" id="offcanvas" aria-labelledby="offcanvasLabel">
<div class="offcanvas-header">
<h5 class="offcanvas-title" id="offcanvasLabel">Offcanvas</h5>
<button type="button" class="btn-close text-reset" data-bs-dismiss="offcanvas" aria-label="Close"></button>
</div>
<div class="offcanvas-body">
Content for the offcanvas goes here. You can place just about any Bootstrap component or custom elements here.
</div>
</div>
<!-- modal -->
<div class="modal" id="modal1" is="dmx-bs5-modal" tabindex="-1">
<div class="modal-dialog" role="document">
<div class="modal-content">
<div class="modal-header">
<h5 class="modal-title">Modal title</h5>
<button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="Close"></button>
</div>
<div class="modal-body">
<p>Modal body text goes here.</p>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
<button type="button" class="btn btn-primary" dmx-on:click="modal1.hide()">Save changes</button>
</div>
</div>
</div>
</div>
<script src="bootstrap/5/js/bootstrap.bundle.min.js"></script>
</body>
</html>
The original Bootstrap components, Off canvas and Modal, do share some style rules. This may be the cause of your problem.
However, the DMXModal, which has been used in my example, should not be burdened with that problem. In other words, have you included:
<script src="dmxAppConnect/dmxBootstrap5Modal/dmxBootstrap5Modal.js" defer=""></script>